Programming

Music Editor Developer’s Log: More Cowbell

I got the soundfont to work, or at least to work well enough for my prototype-creating purposes. It will need some fine-tuning in the future, but if I can manage to actually turn this software into a business, it would be nice to create a custom soundfont for it anyway.

I’m now almost to the point where I can start using this software to actually write some music, but I’ve still got a number of controls and GUI elements (buttons and stuff) to program. I need to add the abilities to do mainly the following:

  • add and delete measures
  • edit note / track variables such as
    • release time (how long it takes an instrument to fade away after it has stopped playing)
    • volume / velocity
    • stereo position (left or right)
  • edit reverb settings
  • save and load files
  • export and load MIDI files (depending on time; this feature isn’t too important yet)
  • export MP3 or WAV files (at least look into it; if this is too time-consuming, it’ll be something to look into in the future)

I think that’s mostly it. And none of that stuff (save for perhaps the last one) should be too terribly difficult to code. So I think I can get it done this week!

After that, I will probably be a bit more secretive as I begin adding the “secret ingredients” which are my amazing world-class AI music generating algorithms, which will be the secret sauce of the business. For that, I will probably have to buy a dedicated server (or VPS), as those algorithms will be executed server-side. That’ll be fun.

Hopefully I’ll also be able to use this editor to actually compose some new tracks this April. I owe my few Patreon subscribers probably around a dozen or so tracks, and I want to get that new album out, which just needs one or two more tracks. And it would just be a good test of the software, even without the AI features, to see what composing with it is like. 122 days left!

Oh, what exactly will constitute success come July 31st? I mentioned earlier that success will mean that the software will either be at a point in which it’s ready (or close-to-ready) to actually market and sell, or in which a working prototype is ready to show to investors. Of course, those possibilities are not mutually exclusive, but at least one must be the case. But what does the latter mean? What will make it “ready” to show?

Anything really, so I can’t lose!

Seriously, though, it will mean that the software should be able to auto-write a complete song (minus lyrics) on its own. That’s melody, chords, orchestration. The algorithms are done, it’s just a matter of making them usable to an end-user and making their output as good as possible.

I’d ideally like the software to be able to compose something with the complexity of a Mozart symphony. That would be the true peak of Parnassus. And I’m positive we’ll have that soon enough. Maybe not by July 31st, but it would certainly be awesome, no?

By S P Hannifin, ago
Programming

Music Editor Developer’s Log: Soundfont Insanity

For the past week, I’ve been trying to give my music editor1 the power of sound. I looked into the new Web MIDI API standards, but those are more for sending and receiving MIDI messages, not playing sound, so that’s no help. (Though it may be something to look into later for other features, of course.)

So instead I’ve been looking into the Web Audio API, which does the trick, and has mostly what I need. Actually, it has everything I need, but not everything I want. I want the sounds to sound as good as possible, which means the instrument samples must loop for sustains (as a MIDI synth would).

First I experimented with MIDI.js‘s implementation of sample playing. With pre-rendered soundfonts, I could easily play samples for all the basic MIDI instruments. Problem with this implementation is that the instruments don’t loop! (Or at the very least, they don’t seem to read in the looping data saved in the soundfont.) Instruments such as strings, which can sustain indefinitely, really deserve some decent looping.2

So I moved on to experimenting with a library called sf2synth.js. I can’t understand the Japanese comments (the developer seems to be from Tokyo), but this implementation seems to load in soundfont files much more completely, and actually reads in and uses the looping data! Woohoo!

But even it has a problem. When I play a note from the Musyng Kite soundfont  (which is the soundfont I’m currently using for experimental purposes) in the Polyphone Soundfont Editor (which is a great piece of software), it sounds great. But when it’s played back in the browser through sf2synth.js, it sounds more bland.

Here is what I think is happening…

If we look at a preset in Polyphone, we can see that it’s actually made up of multiple instruments; below you can see that “Strings Ensemble” is actually made up of 8 layers.

To me, it sounds like sf2synth.js is only playing one of these layers, instead of all of them like a true soundfont player should.

So my mission for this week is to dig into the sf2synth.js code, try to understand how it’s loading and playing sounds from the soundfont file, and try to give it the ability to play all the layers in a preset that it should. Polyphone is open-source, so I can also dig around their code to see how they’re loading in and parsing / interpreting sf2 files.

I probably only want to spend two weeks max on this; if I can’t figure it out after two, I’ll just have to settle for suboptimal sounds and move on. I can always come back to soundfont programming later. It’s more important to get a working prototype finished by the end of July. 129 days left!

By S P Hannifin, ago
Programming

Work on web MIDI editor continues…

Progress on my web-based MIDI editor / animator has been slower than I’d like, but isn’t that always the case? At the moment, I’ve got the basics I want; you can add and delete notes, copy and paste, create and delete tracks, hide and show tracks, and edit track colors. Still need to allow you to add and delete measures though.

But what I want to work on next is the sound; my editor doesn’t actually play any sound yet. I may perhaps try and utilize this javascript soundfont player: https://logue.dev/smfplayer.js/ … of course, soundfonts don’t sound nearly as great as sample libraries, but until someone programs a javascript-based VST host that a browser can use, I’ll have to settle for what I can find. Users should be able to import and export MIDI files anyway.

136 days left until July 31st!

By S P Hannifin, ago
Programming

GUI programming drags on for third week

Checking off some items from the to-do list mentioned in my previous post, my OpenGL GUI panels can now have rounded corners (although they still need a bit of work as they’re not quite as circular as I’d like), they can have shadows of arbitrary size and color, they can have linear and radial gradient coloring, and the text dynamically word-wraps as necessary, with the scroll bar appearing only when needed.

Ain’t that nice? Next I need to work on vertical resizing and making sure multiple panels will overlap correctly when on the screen; a panel should be brought to the front “layer” when clicked on.

Then I will move on to adding the GUI elements I mentioned in the last post.

A fellow programmer on twitch mentioned a GUI library for LWJGL the he had programmed called legui. Looks quite nice so I might play around with it and see if I can integrate any of it with my panels. Another advantage of streaming on twitch: meeting other programmers and discovering new things!

(P.S. I don’t really know if this is my third week of working on it or not, but it’s probably around there. The title of this post is a reference to a film in which people carry in the banner.)

By S P Hannifin, ago
Programming

My GUI programming efforts continue

I’ve been continuing my efforts to program a custom simple 2D/3D engine in Java with OpenGL with which to create a more user-friendly standalone version of my MIDI animator (and hopefully some games in the future). Lately I’ve been focusing on creating a GUI system. In terms of free open-source 3D engines in Java, I don’t much like most of the GUI implementations, so creating my own has been fun. Granted, I still run in to bugs that are tricky to fix while trying to accomplish relatively simple things, but trying to solve such problems is educational and always a bit addicting, like solving a puzzle, especially as I’m doing it for myself.

So here’s my latest progress update. I’ve settled on using STBTT (included in lwjgl) font rendering for the GUI text and the underutilized Nvidia path rendering (included in OpenGL in general) for the panels (and hopefully more in the future, including possible note shapes in the actual MIDI animator). The panels are movable and can contain an arbitrary amount of word-wrapped text. You can scroll through the text, and the panel can include a y-axis scrollbar of arbitrary size. You’ll also notice that the text fades in and out at the top and bottom, a nice little touch in my opinion that I probably spent too long programming:

I’ve also been streaming my programming efforts on twitch, which is also addicting. It turns what is usually a pretty solitary activity into something at least a little more social, and it’s fun watching the viewer count tick up now and then. Shout out to twitch user Subtixx who has even offered helpful points and advice allowing me to fix problems my eyes don’t catch (such as null-checking the darn scrollbar!); many thanks Subtixx!

My current to-do list includes (but is not limited to):

  • allow gradient coloring on the panels and text
  • allow rounded corners on the panels
  • allow panels to have shadows (render a semi-transparent panel behind)
  • allow panels to be resized dynamically
  • allow panels to be closed (and maybe shrunk?)
  • allow panels to contain some select common GUI elements, such as:
    • buttons
    • text fields (short and long)
    • radio buttons
    • check boxes
    • sliders (continuous and quantified)
    • menus (pop-up and drop-down)
    • tabs on the top or side of panels (which perhaps are just specialized buttons?)

So there’s still quite a bit to do. But I must do it, for this is my quest.


In other unrelated news, a 4th album from symphonic / power metal band Ancient Bards is finally happening! I love this band and I can’t wait for their new album!

By S P Hannifin, ago
Programming

Font rendering: stb oversampling vs NV path rendering

I told you I wanted to try stb oversampling in my last post, and I did! Here is the result (stb 2×2 oversampling on top, nanovg in middle (which is based on stb anyway, but no oversampling), and NV path rendering on bottom, screen size then 5x zoom):

So stb oversampling definitely looks the best, although it is still pretty fuzzy. And moving it around by subpixels looks very decent; it doesn’t get very much of the “shimmer” effect. (Still a bit, but not much to be bothersome.)

Here’s stb oversampling (top) vs NV path rendering (bottom) with a bigger font size (screen size then 2x zoom):

Here, I think NV path rendering looks better; it’s definitely less fuzzy. (The trade-off is that it does suffer from more “shimmer” when translated by subpixels, but it doesn’t bother me too much.)

You can also see that NV path rendering is able to utilize proper kerning: the ‘e’ is slightly below the capital ‘T’, as it should be. Each letter isn’t being drawn on a textured quad, so overlapping is trivial. (Well, for the end library user, at least.)

So, I think that completes my foray into font rendering for now. I’m too lazy to make bitmap fonts at the moment; stb oversampling will work for smaller fonts for now. Time to continue on to other GUI elements. I will try to design the GUI system in such a way that it will be able to utilize any font rendering system should I wish to create bitmap font support in the future.

By S P Hannifin, ago
Movies

More font fun and other random stuff

Font rendering in OpenGL

Haven’t done so much programming in the past week, but I did try rendering fonts with NanoVG (in lwjgl). Unfortunately it’s really not much better than just using OpenGL’s NV path rendering extension. Small fonts look slightly better, but not really good enough for me to want to use them. See the example below, a zoom-in of an 8-pixel high rendering of the font “Verdana”, NanoVG rendering it on top, NV path rendering below. NanoVG is better, but it’s still way too fuzzy to look any good.

So I might just use bitmap fonts for small text; I can’t see any alternative. (Bitmap fonts basically means loading in each letter as a pre-rendered picture and plopping it on the screen. The disadvantage is that they don’t look very good when resized or positioned between pixels, but they’ll at least be guaranteed to render small fonts clearly and crisply.) I’ll continue to use NV path rendering for larger fonts or fonts that need to be animated more dynamically or rendered with 3D perspective. I do want to try using oversampling with stb for small fonts before I move on from font rendering and further develop a GUI system; the demo doesn’t look too bad.


Kanopy film streaming service

I just realized our local library offers free access to the film streaming service Kanopy, and they’ve actually got a decent selection. Not the latest blockbusters, but some good foreign and classic cinema. (They’ve even got The Red Pill, the controversial anti-radical-feminist documentary which Netflix refuses to stream. (Though they did just recently finally add the DVD to their catalog.)) Since it’s free (for library card holders of participating library systems), we’re limited to 10 streams per month, but the streaming quality is decent. It’s not full HD (at least not on my PC), but it’s better than DVD quality. Interestingly they also allow you to embed videos. Here’s “Kumiko the Treasure Hunter”, which I really enjoyed (which I guess you won’t be able to see without an account, haha):

Interesting indeed!


Some metal music

Finally, I recently discovered the band Elvenking. I couldn’t quite get in to some of their older work, but their last album, released last year, is quite catchy. Disregarding the bizarre sense of fashion metal band members tend to share, this song is some power metal awesomeness:

I also came across the latest album from the symphonic metal band Leaves’ Eyes, and found it to be quite catchy as well. Love the use of choirs, the female lead’s operatic voice, and the cheesy fantasy lyrics. (I’m not a big fan of the growl singing, it sounds so gross and ugly and demonic, why is it so popular? It sounds so awful, so unmusical. Bah!)

OK, that’s all for now.

By S P Hannifin, ago
Programming

Font programming fun

So I’ve started working on GUI elements for my little engine that could. My quick list of things I’d like to include in a GUI system:

  • ability to create a menu bar
  • ability to create windows / panels with drop shadows
  • ability to create scroll bars for window content
  • ability to create buttons and button animations (hover, click)
  • ability to create text boxes and text fields

I started working on text today, i.e., font rendering. The NV path rendering extension actually does a pretty good job with larger text:

It’s not so good with smaller text:

Look at that ugly subpixel rendering:

Bleh! So I’ll have to figure out something else for smaller fonts. I’ll probably look into the Java AWT package, or perhaps look into NanoVG, especially since it looks like lwjgl already has bindings for it. (Maybe it already has everything I want?)

By S P Hannifin, ago
Programming

MIDI animator version 2 wishlist…

As I blogged a couple months ago, one of my goals for 2018 is to program a very simple 3D engine. I’ve been wanting to program a simple adventure game for a few years now, but the 3D and 2D engines I’ve played around with just don’t have quite the features I want, so I’m attempting to create my own little game engine with the Lightweight Java Game Library (lwjgl).

Here’s what I’ve got so far: I can render a textured box and some resolution independent vector graphics including true-type fonts, which can intersect with the box!

OK, so nothing special yet.

Anyway, another thing I’d like to create with lwjgl is a new version of my MIDI animator, using vector graphics (instead of custom shaders) to represent the MIDI notes. (On a side note, I’m using the OpenGL “NV path rendering” extension for vector graphics support, which was created by Nvidia. Does that mean it’ll only work on their GPUs? I don’t know.) This should easily allow many more possible note shapes to be created.

Here’s my current wishlist for “MIDI animator 2.0” (I should also come up with a more interesting name):

  • Should be able to run standalone (that is, users shouldn’t have to download and compile the code to use it)
  • A graphical user interface (GUI) that makes it easy to:
    • load MIDI files
    • change note styles / shapes / colors / animations
    • a scroll bar to make it easier to scroll through the MIDI
    • save files
  • Ability to add title / text animations (instead of having to do this in post)
  • Ability to change / animate background
  • Ability to load in an MP3 to sync with the visualization
  • Ability to add an MP3 visualization, such as one of those pulsing frequency bars or something
  • Ability to export a movie file automatically (if I can figure out how; I’ll save this for last)

Anything else I should include or look into?

So, yeah, this is probably what I’ll be working on in my free time for now. I’ll probably start working on a GUI system next as I’d need to do this for a future game anyway.

Hopefully I’ll get back to actually writing some new music afterwards.

By S P Hannifin, ago
Programming

Game programming…

Last year I learned the basics of jMonkeyEngine, and though I intended to create a 2D game with it, I only ended up creating the MIDI animator mentioned in my last post. But for a tile-based 2D game, it’s really not ideal. It’s certainly capable of producing such a game, but as an engine, it’s not really designed for it, and I realized I’d have to spend a considerable amount of time just programming a custom framework for such a project.

So last month I began learning LibGDX, which is certainly better suited for the sort of tile-based 2D adventure game I’d like to create.

So here’s what I’ve got so far (the art is temporary, simply a free tile set I found out there, though I think I’ll be aiming for tiles that are 16×16 pixels as well, so this represents the size I’m hoping for):

screenshot

Doesn’t look like much, but I’m still learning and programming the basics I’ll need before I actually start programming the actual game play. What I’ve learned and/or programmed so far includes:

  • loading and displaying a tilemap (a .tmx file from Tiled)
  • dynamic window resizing (or resolution changing) without completely screwing up the aspect ratio (trickier than it may sound)
    • basically it will always render with a resolution of 1280×720, which is then rendered as a texture to whatever resolution the user sets; handy for Android devices which may feature differing resolutions which cannot be changed
  • using Ashley as an entity framework
  • using Box2D for character movement and collision detection
  • animating the character as he walks (much easier with LibGDX; I had to program a custom shader with jMonkeyEngine)
  • real-time A* pathfinding (hope to use it with NPCs)
  • text (as you can see at the bottom there) to be used with menus, dialog, etc. (nice little font called Munro) (this is also much easier with LibGDX)
  • custom shader to make object tiles “light up”

Still lots to program, but I reckon that’s an OK start. The to-do list before I can actually begin programming the actual game-play logic includes:

  • have interactable objects “light up” as the player nears them, along with a small pop up menu with available actions (pick up, talk, examine, activate, etc.)
  • design and implement a menu panel / status bar (along the black panel along the bottom)
    • this menu will switch to a “dialog mode” when the character is talking with someone (as in The Secret of Monkey Island, for example)
  • pause menu with save, load, quit, settings, etc.
  • camera movement for when the player walks off the screen
  • loading another tilemap for when the character leaves a village or enters a house, for example
  • I need to try sticking an NPC in there and having it play through some scripted material, utilizing the aforementioned A* algorithm as necessary
  • real-time combat system
    • cast a spell or swing a sword to injure attacking enemies (the combat will be simple in this game, nothing fancy)

That’s all I can think of at the moment, but that’s plenty, isn’t it?

The game itself will be pretty short; I’ve tried to keep it short on purpose since even a short game is a lot of work, especially when you’ve never done it before and don’t have a pre-existing framework your familiar with to use. If I manage to actually succeed in this endeavor, I hope it will be easier to create new adventure games and/or RPGs in a similar style without having to reprogram these basics again, at least not from scratch.

The tentative title for the game is Memory of a Thousand Kings, which began as a half-plotted fantasy novel, but I think it will work better as an adventure game.

By S P Hannifin, ago