Yes, but that says more about the chosen data structure and how well it is suited for one but not the other use case.

How about the following column orientated data structure supporting row access: Each column is an arraylist, the hashmap the index. Using the hashmap you find that you want to read row #1234, hence you access each of the column-lists with arraylist.get(1234). Then you have equal access times like in a row storage, where you get the rowid from the index, access the row from a single arraylist and then the row is a list of column values.