If I correctly interpreted the extend of your alignment, you've about half way done. You checked how the wheels are sitting in their dropouts, the fork is fine and the rear end is shifted ~3mm.
Here's what you need to check next: is the seat tube sitting on the same plane as the head tube? In other words, if the frame was sitting next to a wall would the head tube and the seat tube be parallel to the wall and be the same distance away from the wall? Then are the wheels also parallel to the wall. Ideally you should be able to cut a bike in half down the middle (like Wile E Coyote running though a buzz saw) and have the same amount of bike on both sides. If the head tube is off axis from the seat tube, you'd have more BB on one half and more seat on the other half.
Does that make sense? The big question is "how much axis deflection is too much?" That's a difficult question. I'm not great at geometry, but if you're off 1mm (1 degree?) at the top of the fork crown, how much does that increase to at the ground? Is that too much to live with?
In the late 80's I had a 531 tubed Nottingham made frame that was twisted like I just described. It never gave me issues nor had I found a high speed wobble. But at a mechanic school we put it on a frame table to learn some alignment techniques and discovered the axis issue. After adjustment it was a different bike. It rode so much better.
If you have access to a frame table it will be easy to verify the alignment. If you have a Park F.A.G.-2 and other tools
you could do this procedure. A not so quick (it could take a couple of hours depending on distractions) and dirty way to see is to put the bike in a stand next to a large window made from one sheet of glass. Glass isn't always perfect, but it's closer than drywall or brick walls.
Tools you'll want or need:
=A good tape measure
=a 3' or larger carpenter's level
=a small level that could fit on the head tube (if possible find one that will avoid the lugs if it is a lugged frame)
=a level that will fit on the seat tube
=masking tape
=a fine point Sharpie (ultra fine if you can swing it)
= a couple of window shims
=about 10' of string that doesn't stretch
How all this comes together:
Take the carpenter's level and check the window. If it is plumb to the ground and has no gaps along the length of the level then you're stylin'. If there's defects in the window where there's a gap between the level and the glass, put a small piece of tape on the gap so you can avoid that spot when measuring.
Take the string and tightly tie between the front axle and the bottom bracket spindle on both sides of the bike. Don't get fancy with the knots, but they can't slip and the string needs to be uber-tight. Measure the distance from the BB center to the front axle center. It needs to be the same on both sides of the bike.
Set the bike stand next to the window as close as you can and still be able to use the tape measure. The bike stand needs to be anchored solidly so it doesn't shift away from the window between measurements. If you have a shop-grade Park stand with the heavy base that's about as solid as they come. Otherwise you'll want something like a sandbag to set on the feet of the stand.
Once you've referenced the plumb of the window with the seat tube level and head tube level (if it's off plumb and the bubble is partially on a line you can still do this. With the Sharpie put a small mark on the edges of the bubble for the new center) you need to match the plumb of the seat tube. We want the seat tube to be the zero reverence to the rest of the frame since it's the hardest to adjust. Put the seat tube level on the seat tube. IF it's a match, great, otherwise you'll need the window shims to gently shim the bike stand. Get the seat tube parallel to the window and then rotate the arm of the stand until the seat tube is plumb.
If you have the levels on the head and seat tubes and the bubbles referenced on the window are within the same marks from the window you're golden. If the head tube level is off then you know it's the problem.
The other measurements you can verify are the wheel centers. If the head tube and seat tube are both the same distance (measure to center of the seat & head tubes) from the window then the wheels should also be the same distance from the window. (the rear would be ~3mm off, right?) For the rear wheel you can measure from clock points to the window and see if it is actually sitting in the frame centered. Make sure the wheel is dished and true. If the wheel is out of dish or out of true, put a piece of tape on the rim and make all measurements to the window from that mark. Just rotate the wheel to make the measurements.
The front wheel should also be parallel to the window, unless the head tube is off axis from the seat tube. Then the top of the wheel would be closer or further away from the window.
So you just wrestled with the bike to discover it's off axis. What to do? Do you have the tools to tweak the frame back to axis? If you don't, I don't know what to say other than you're going to have to live with it and avoid washboard descents at high speed.