I have taken my first steps into fear, uncertainty and doubt: gone live with a global MMO that uses Linux+Java and the rest is my own code. The distributed 100% read uptime async. to async. database ROOT is now working as intended. Performance is stellar and stability is good.
The next logical step is 100% write uptime, which if you ask the internet, is the same thing as Paxos or impossible. If you try to figure this out for long enough; you'll finally come across that in Paxos, everytime you ask for a value; you ask atleast a majority of computers, which means you still have a similar bottleneck as any SQL database upon read!? Or did I get this wrong? This would break the local read efficiency gain of my 100% read uptime solution for something that is not 100% write uptime.
So this is where I'm at now. I need reconciliation and that seems easy to get with a chain, merkle tree, 1 hour of "memorypool" RAM quarantine before stuff hits the disk, "synchronized UTC time ordered reconciliation" with "id order as time collision fallback" and some not yet finalized communication protocol (think rsync, but over async. HTTP).
I then realized since my database always generates random id's (I don't need leader election) consensus corruption can be avoided simply by forcing each node to create id's that begin with certain numbers (in my case since I only have two cluster nodes it could be even and uneven or 1-4 and 5-9 for example) allowing for reconciliation even if partitions keep working separately of each other for some time. This also has the additional benefit of working with even amounts of nodes, another thing Paxos can't do.
But then, over lunch, my friend pointed out a flaw in my "memorypool of 1 hour to avoid uneccessary ordering problem before the transactions hit the disk".
What happens if the cluster is partitioned and one partition crashes... well then 1 hour of half the transactions are lost. In my case this means there must be two concurrent problems:
1. Fiber Partition: My two nodes can't reach eachother but are still reachable for others independently of eachother.
2. Power Failure: All computers of one node must crash at the same time.
I think it's worth that risk, don't you?
So I'm now fooling myself that I have found a good alternative to Paxos that is only sensitive to two very improbable failures that would be even more improbable to happen at the same time.
It's not 100% write uptime, but it's good enough, or at least better than Paxos for my needs.
If you would be so kind to help me prove myself wrong I would be infinitely grateful.
PS. By node I mean many computers behind one IP, all part of the consensus. Also my systems never collide on data: two users never access the same data at the same time from different clients DS.