Wow, the Sakura open source MP3 player kit opens up all kinds of possibilities for installation projects.
For around $30 in parts and a good amount of patience, you can have a completely open source and hackable mp3 player ready to go. It can be modified to accept serial commands, be embedded in an art project, used as the voice of your next smart talking robo-sidekick, or filled with music and used as is. Put in whatever size card you want, up to the theoretical limit of the MMC format! All the source and schematics are here for free as part of the Creative Commons. I have kits available if you don’t feel like scavenging for the parts yourself.