melody project

TuneSage progress update 12

I’ve recently been working on a “pop” style for melodies, but they still sound a bit generic at this point. (There are plenty of samples in the video below.) With melodic styles having a lot of overlap, I’m not quite sure how or where I’ll end up drawing distinctions. But I’m less focused on delineating styles at the moment, and more focused on broad capabilities, making sure new things it learns remain compatible with what it’s already learned. For now, it’s got two new “forms”: a 4-bar phrase that repeats, and a 2-bar phrase that repeats twice followed by a 2-bar closing. It’s also does some minor syncopation now and then, and I’ve added a setting that allows users to set the “note density” to fine, medium, or coarse. (Fine means: more syncopation, more 16th notes, sounds better at slower tempos. Coarse means the opposite: less syncopation, no 16th notes, sounds better at faster tempos.)

I also added “theme blocks”, the idea being that one can write a mockup for a whole song by stringing together multiple themes.

Still much work to do, including what I mentioned last time: allowing the user to “lock in” part of a melody and have TuneSage regenerate the rest. I need more forms, especially ones longer and shorter than 8 bars. I’d like to allow the user to generate chord progressions separate from the melody, or supply their own chord progressions.

Even with such a lot of remaining work to do, I’d really like to release the app as soon as possible, despite its initial shortcomings, just to see if anyone out there is even interested in paying for such a thing, and if so, get some real user feedback. I’m aiming for releasing it at the end of July or early August. I’m tempted to continually put it off later and later as I add more capabilities, but I could probably continually do that for years, and I’ve already spent way too many years just getting my initial music theory ideas to this current state. Stay tuned!

(P.S. I’m also making the devlog videos on YouTube to help me practice presenting, as I don’t think I am very good at it.)

By S P Hannifin, ago

TuneSage progress update 11

Haven’t blogged in a while! Does anyone read blogs anymore anyway?

Also haven’t made an update about TuneSage in a few years. But I’ve been working on it on and off, trying a lot of approaches and doing a lot of experiments that ultimately failed. Lots of trial and even more error.

But I finally have a completely overhauled algorithm that is both extremely fast and much easier to iterate on (unlike previous versions), so I’m excited and finally feel like I’m making good progress once more (despite the emergence of a vocal anti-AI crowd that didn’t much exist 10+ years ago).

Anyway, I’ve just made a YouTube video demonstrating TuneSage as it is at the moment:

As I mention in the video, the three main things I want to add before launching the product:

  • Melodic styles (right now the melodies are purposefully very random and generic sounding)
  • The ability to write melodies more or less than 8 bars long, and the ability to string melodies together for larger forms
  • The ability to edit individual notes or chords, and the ability to “lock” notes or chords in place while re-generating the rest

Plus, I’ll probably move the chordal accompaniment to a separate track so that the user can choose different instruments for both.

Still, the app will focus only on generating melodies with chords at first. I’ll work on expanding the accompaniment generation abilities after the app is actually launched.

(P.S. As of this writing, tunesage.com is down because I switched hosts and still haven’t reuploaded the landing page… I should do that soon.)

By S P Hannifin, ago

TuneSage progress update 5

It’s been over a year since my last TuneSage update, but work has been progressing. Work on the backend was slow and challenging; I spent some time going down quite a few dead-ends. But it’s actually generating melodies now, so I should be able to release some output samples sometime soon, fingers crossed.

I signed up for this year’s Y Combinator’s Startup School once again. I did it once back in 2019, but other than making a landing page (TuneSage.com), my progress was sparse. I just needed a lot more time than I imagined to make progress with the backend.

Startup School’s Course Guide says: “If you haven’t launched yet, make it a goal to launch during the program and get your first users!”

OK, I guess I’ll make it a goal then! Granted, I have repeatedly failed at making enough progress to launch whenever I have set it as a goal, but one must keep trying I suppose.

My question is: how many features do I need to launch with? For better or worse, my current plan is to just launch the product as a basic melody generator to start with.

So what do I need to do to launch?

  • Prepare the backend
    • Train the AI on more melodies (using public domain melodies)
    • Generate melodies in a variety of styles (these will be basic to start with)
  • Add at least some simple chordal accompinement features to frontend
    • e.g. root notes, arpeggio patterns, alberti bass, etc.
  • Overhaul frontend design (lots of tedious web design) and finalize
  • Figure out what soundfonts TuneSage will use
  • Figure out deployment and version control (honestly this can probably wait until after launch, but not too long after)
  • Create user account system
    • Create new account
    • Confirm email (if necessary)
    • Log in / out / reset password
    • Edit optional personal info
    • Usage stats
    • Terms and conditions
  • Register company
  • Find some payment system to use
  • Allow for a trial period (and decide exactly what that consists of)
  • Install some analytics system (so I can keep track of user engagement or whatever)
  • Stress testing? (In my experiments with “trovedex”, the database kept going down; I really don’t want that to be a problem. Anyway, I can always do invite-only if the system is too stressed, but that would be a good problem to have)
  • Launch! Update front page with information

Is that it? Am I forgetting anything?

Of course, there are many more features I’d still like to add; AI can be used for a lot more than just generating melodies. But it’s a starting point, and melodies are the one area I think other AI music systems struggle with the most at the moment.

So how long will all that take? Can I finish it in a few weeks? Startup School lasts for 7 weeks, so if I can do it in 3 or 4 weeks, that would be awesome. Considering how long things have taken me in the past, however, it will probably take me… 12 years. But for now let’s daydream:

  • Week 1: Finish backend and overhaul frontend
  • Week 2: Soundfont and user account system, start releasing samples
  • Week 3: Register company, install payment and analyctics systems
  • Week 4: Set up trial, stress testing, front page update, and launch!

That’s probably wishful thinking, but it’s better than nothing.

By S P Hannifin, ago

Music Editor Developer’s Log: More Cowbell

I got the soundfont to work, or at least to work well enough for my prototype-creating purposes. It will need some fine-tuning in the future, but if I can manage to actually turn this software into a business, it would be nice to create a custom soundfont for it anyway.

I’m now almost to the point where I can start using this software to actually write some music, but I’ve still got a number of controls and GUI elements (buttons and stuff) to program. I need to add the abilities to do mainly the following:

  • add and delete measures
  • edit note / track variables such as
    • release time (how long it takes an instrument to fade away after it has stopped playing)
    • volume / velocity
    • stereo position (left or right)
  • edit reverb settings
  • save and load files
  • export and load MIDI files (depending on time; this feature isn’t too important yet)
  • export MP3 or WAV files (at least look into it; if this is too time-consuming, it’ll be something to look into in the future)

I think that’s mostly it. And none of that stuff (save for perhaps the last one) should be too terribly difficult to code. So I think I can get it done this week!

After that, I will probably be a bit more secretive as I begin adding the “secret ingredients” which are my amazing world-class AI music generating algorithms, which will be the secret sauce of the business. For that, I will probably have to buy a dedicated server (or VPS), as those algorithms will be executed server-side. That’ll be fun.

Hopefully I’ll also be able to use this editor to actually compose some new tracks this April. I owe my few Patreon subscribers probably around a dozen or so tracks, and I want to get that new album out, which just needs one or two more tracks. And it would just be a good test of the software, even without the AI features, to see what composing with it is like. 122 days left!

Oh, what exactly will constitute success come July 31st? I mentioned earlier that success will mean that the software will either be at a point in which it’s ready (or close-to-ready) to actually market and sell, or in which a working prototype is ready to show to investors. Of course, those possibilities are not mutually exclusive, but at least one must be the case. But what does the latter mean? What will make it “ready” to show?

Anything really, so I can’t lose!

Seriously, though, it will mean that the software should be able to auto-write a complete song (minus lyrics) on its own. That’s melody, chords, orchestration. The algorithms are done, it’s just a matter of making them usable to an end-user and making their output as good as possible.

I’d ideally like the software to be able to compose something with the complexity of a Mozart symphony. That would be the true peak of Parnassus. And I’m positive we’ll have that soon enough. Maybe not by July 31st, but it would certainly be awesome, no?

By S P Hannifin, ago

Music Editor Developer’s Log: Soundfont Insanity

For the past week, I’ve been trying to give my music editor1 the power of sound. I looked into the new Web MIDI API standards, but those are more for sending and receiving MIDI messages, not playing sound, so that’s no help. (Though it may be something to look into later for other features, of course.)

So instead I’ve been looking into the Web Audio API, which does the trick, and has mostly what I need. Actually, it has everything I need, but not everything I want. I want the sounds to sound as good as possible, which means the instrument samples must loop for sustains (as a MIDI synth would).

First I experimented with MIDI.js‘s implementation of sample playing. With pre-rendered soundfonts, I could easily play samples for all the basic MIDI instruments. Problem with this implementation is that the instruments don’t loop! (Or at the very least, they don’t seem to read in the looping data saved in the soundfont.) Instruments such as strings, which can sustain indefinitely, really deserve some decent looping.2

So I moved on to experimenting with a library called sf2synth.js. I can’t understand the Japanese comments (the developer seems to be from Tokyo), but this implementation seems to load in soundfont files much more completely, and actually reads in and uses the looping data! Woohoo!

But even it has a problem. When I play a note from the Musyng Kite soundfont  (which is the soundfont I’m currently using for experimental purposes) in the Polyphone Soundfont Editor (which is a great piece of software), it sounds great. But when it’s played back in the browser through sf2synth.js, it sounds more bland.

Here is what I think is happening…

If we look at a preset in Polyphone, we can see that it’s actually made up of multiple instruments; below you can see that “Strings Ensemble” is actually made up of 8 layers.

To me, it sounds like sf2synth.js is only playing one of these layers, instead of all of them like a true soundfont player should.

So my mission for this week is to dig into the sf2synth.js code, try to understand how it’s loading and playing sounds from the soundfont file, and try to give it the ability to play all the layers in a preset that it should. Polyphone is open-source, so I can also dig around their code to see how they’re loading in and parsing / interpreting sf2 files.

I probably only want to spend two weeks max on this; if I can’t figure it out after two, I’ll just have to settle for suboptimal sounds and move on. I can always come back to soundfont programming later. It’s more important to get a working prototype finished by the end of July. 129 days left!

By S P Hannifin, ago

Y Combinator

Y Combinator looks like something I’d really like to do. Looks like it would be perfect for the symphony generator app I’ve been working on. As they say on their site:

Twice a year we invest a small amount of money ($120k) in a large number of startups.

The startups move to Silicon Valley for 3 months, during which we work intensively with them to get the company into the best possible shape and refine their pitch to investors. Each cycle culminates in Demo Day, when the startups present their companies to a carefully selected, invite-only audience.

The deadline for their upcoming Winter 2019 cycle is October 2nd, less than two months away. Unfortunately I’m not so sure I can finish a working prototype by then. Worth a try though. Might have to wait until their next cycle after that. Regardless, this looks like something to aim for!

By S P Hannifin, ago

More new music and what happened to the melody generator

I uploaded a new piece of music to YouTube earlier today called The Stormbringer’s Apprentice:

I write in the description:

This piece provides the themes for a villain called Stormov, from my book Insane Fantasy. He’s an apprentice for the mysterious “Stormbringer”, and helps to entice new recruits for his master’s evil plans. His themes are mostly dark, but there is a hint of adventure in there as well, as he lures potential helpers with temptations of worldly power.

Also, this is my first video to feature my own attempt at programming my own custom “music animation machine” programmed in Java with jMonkeyEngine. The frame rate is not as smooth as I’d like it to be (I’m still screen-capturing it), but it’s not horrible at least. Anyway, I think I’ll enjoy playing around with it. Obviously it’s inspired by Stephen Malinowski’s work, as I’ve been using his decade-old “Music Animation Machine” program for a long time now.

As mentioned above, this piece is animated with my own custom “music animation machine.” I may release the source code for it at some point for anyone who’s interested in it, but you’d have to download jMonkeyEngine to compile it and run it yourself; I’m not really interested in making into a standalone program right now, as I hope to continue fooling around with the code. The code is also sloppy and contains features I never finished programming, as it’s all part of an ongoing MIDI sequencer project. Also, it doesn’t actually even play MIDI files… it only makes the shapes and animations based on a MIDI file, which is all I need it to do as I sync videos with tracks recorded from Overture in Windows Movie Maker before adding titles and uploading to YouTube.

In case you missed it, last week I posted this track called Storybook Overture:

So that makes five tracks completed so for my upcoming album (which I have no idea what I’ll name yet): Lullaby of the Westwind Woods, The Storm Cometh, A Stargazer’s Lullaby, Storybook Overture, and The Stormbringer’s Apprentice. Together they amount to over 20 minutes, so I’m over 1/3rd finished the album!

In other news, I’ve had several people email me over the last few months asking about whatever happened to the melody generator. Well, I was contacted by a shady underground group that secretly controls the world, and they warned me that the world simply is not ready for something so powerful.

Actually, the project just got to be too frustrating. It was getting enough interest that people were thirsty to learn how it worked, yet not enough interest to fund a Kickstarter, and I just didn’t (and don’t) have the time to give it the attention it needs. I of course haven’t given up on it, but it’s on the back-burner for now until I can get my life sorted a bit more. In the meantime, I’d just rather spend my free time writing music and books and programming games, as those projects tend to actually generate some income. Not much, but something. So I have absolutely no idea when I’ll be able to get back to working on the melody generator. Could be later this year, could be two or three years down the road.

I have an email list here if you’d like to be updated when I actually return to the project: Melody Generator News.

By S P Hannifin, ago

Trying a Kickstarter sometime soon…

I’ve been in a bleak creative rut lately. I feel like doing nothing creative at all. I’ve had no inspiration, interest, or will to do anything. Go to work, come home, eat, and waste time watching movies and surfing the web. A dull, boring, purposeless, vacuous existence.

So in an effort to give myself a bit of purpose, I’ve been working on putting together a Kickstarter campaign called “Give me some money to smile again.”

OK, it’s not called that… it’s actually fundraising for that book on melody that I’ve been writing on and off for years. The Secrets of Melody. It details my theory of melody, including how my melody generators work. Basically, if I can raise the funds, I can finish it by March 2016. Of course, I have hardly any online following, so I know it’s a long shot. Seems worth a try though, especially as I feel no great desire to do much else at the moment.

And if I fail to raise the funds, I’m going to go ahead buy an Oculus Rift dev kit and start having some fun with it to make myself feel better. So it’s a win-win for me. And I might buy one regardless because I’ve been dreaming of VR and I’ve got some fun ideas for projects I’d like to try with it. But if there’s enough interest for my book on melody, I’d love to make that a priority.

Anyway, I just have to make the Kickstarter video, which is hard because I’m the self-conscious type who hates looking at a video of himself. But I’ll try anyway.

So look out for that sometime this week.

By S P Hannifin, ago

February happenings…

Haven’t gotten much done this week. I’m adjusting to a slightly new work schedule with a bit more hours (which is good for the paycheck), plus some chest congestion has been keeping me from getting to sleep, and I keep waking up with headaches. So I haven’t been very good at finding large chunks of time to do any very focused work.

Anyway, I’ve been a bit torn lately between competing interests: should I work on my fiction writing, or my music software? Whenever I commit to one, I feel the pull of the other. Ultimately, though, I think I’m going to focus on programming my music software for a while. I’m excited about its potential, plus it would be awesome if I could turn it into a business and be self-employed with it. Not that I’m not arrogant enough to believe I could do the same with writing with more work and patience, but music software that’s like nothing else on the market (as far as I know) probably stands a better chance.

By S P Hannifin, ago