Software to roll-your-own Open Access journal are now fairly easy to get hold of. OJS is available from the Public Knowledge Project as Free Open Source Software, Annotum builds on top of Wordpress and there are forthcoming projects such as Faculty that also broaden the field. However, there is also a great deal of concern, no doubt hyped by publishers who are keen to discredit the homegrown OA community, that these journals are short-lived and disreputable.
I've already written elsewhere, and in detail, of the ways in which academic capital manifests itself (or is cloaked) within academic publishing systems and how this should be the determining factor for publication, rather than outmoded tradition. I want today to plug a resource that can help an author decide on the longevity of an OA journal and decide upon the safety of publishing in it.
The system that I want to highlight is called The Keepers Registry and it is a database, frequently updated, that shows the archival and digital preservation provisions of a journal. While it's not specific to Open Access journals (and it's worth checking this out for *any* journal in which you're considering publishing), it's of particular use where there might be concerns over the journal's longerm stability. For example, if you search for Orbit: Writing Around Pynchon, you'll see that the journal is archived by CLOCKSS and LOCKSS and so, in the event of my untimely demise beneath a bus, the journal and its contents are safe.
The other aspect that it's worth checking (via a quick visual inspection) is whether the journal you're considering issues DOI numbers to its articles. If it does, it should have digital preservation systems in place (this is part of the CrossRef contract), but it's still worth checking. The DOI numbers mean that, even if the journal went down, the DOI numbers would remain active and would be redirected to the archived version.