Flightradar24 — how does it work? Part 2, ADS-B protocol

в 21:04, , рубрики: ADSB, Flightradar24, IT-стандарты, python, radio, SDR, signal processing, инженерные системы, Научно-популярное, распределенные системы

I’m going to have a guess and say that everyone whose friends or family have ever flown on a plane, have used Flightradar24 — a free and convenient service for tracking flights in real time.

image

In the first part the basic ideas of operation were described. Now let's go further and figure out, what data is exactly transmitting and receiving between the aircraft and a ground station. We'll also decode this data using Python.

History

It should be obvious that airplanes data is not intended only for users, who want to see it on their smartphones. This system is called ADS–B (Automatic dependent surveillance—broadcast), and was designed for the automatic transmitting of the aircraft information data to the control center — different parameters, as coordinates, speed, heading, altitude, and others, are sending. Previously, the dispatcher could see only a point on the radar screen. And it became definitely not enough when number of airplanes drastically increased.

Technically, ADS-B consists of a transmitter inside the aircraft, that periodically sends information data frames at a relatively high frequency of 1090 MHz (there are some other modes, but they are not so interesting to us, because the coordinates are transmitted only here). Of course, there is also a receiver somewhere at the airport, but for us, as for users, our own receiver is more interesting.

Just for comparison, the first such a system designed for ordinary users, the Airnav Radarbox, appeared in 2007, and costed about $900, and about $250 per year costed a subscription to their network services (the main idea of this system is to collect and share data from many receivers, a standalone receiver is relatively useless).

Flightradar24 — how does it work? Part 2, ADS-B protocol - 2

Now, when RTL-SDR receivers have become much more available, a similar device can be made for $30. It can be found in the first part of the article, and we will go further and describe the protocol itself — let's see how it works.

Receiving a signal

First, we need to record a signal sample. The whole signal has only 120 microseconds length, and to see its details in a good «resolution», its better to have SDR-radio with at least 5MHz sample rate.

image

After recording, we are getting a WAV-file with a 5000000 samples/sec sampling rate, 30 seconds of such a record has about 500MB size. Of course, it’s useless to listen it with your favorite media player — the file does not contain a sound, but a directly digitized radio signal itself — this is exactly how Software Defined Radio works.

We can open and process this file with Python. Those who wish to do the experiment on their own, can download a sample from this link.

Lets open a file and see, what is inside.

from scipy.io import wavfile
import matplotlib.pyplot as plt
import numpy as np

fs, data = wavfile.read("adsb_20190311_191728Z_1090000kHz_RF.wav")
data = data.astype(float)
I, Q = data[:, 0], data[:, 1]
A = np.sqrt(I*I + Q*Q)

plt.plot(A)
plt.show()

Result: we see some 'impulses' over the noise.

Flightradar24 — how does it work? Part 2, ADS-B protocol - 4

Each «impulse» is a signal, whose structure is clearly visible if we increase the resolution on the graph.

Flightradar24 — how does it work? Part 2, ADS-B protocol - 5

As we can see, the picture is fully consistent with its description above. We can now process this data.

Decoding

First, we need to have a bit stream. The signal itself is encoded with a manchester encoding:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 6

From the half-bite differences we can easily get real «0» and «1»'s.

    bits_str = ""
    for p in range(8):
        pos = start_data + bit_len*p
        p1, p2 = A[pos: pos + bit_len/2], A[pos + bit_len/2: pos + bit_len]
        avg1, avg2 = np.average(p1), np.average(p2)
        if avg1 < avg2:
            bits_str += "0"
        elif avg1 > avg2:
            bits_str += "1"

The structure of the signal itself looks like this:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 7

Lets see the fields in more detail.

DF (Downlink Format, 5 bits) — defines the type of message. There are several types of them:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 8
(page source)

We are only interested in the DF17 type, because only this one contains the aircraft coordinates.

ICAO (24 bits) — is an unique international aircraft code. We can check the aircraft by its code on this website (unfortunately, the author has stopped updating the database, but it is still relevant). For example, for the 3c5ee2 code we can have the following information:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 9

DATA (56 or 112 bits) — is the data itself, which we will decode. The first 5 bits of data is the Type Code field, which contains the subtype of the data being stored (not to be mixed with DF field). There are a lot of such types:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 10
(table source)

Let's look at some examples.

Aircraft identification

An example in binary form:

00100 011 000101 010111 000111 110111 110001 111000

Data fields:

+------+------+------+------+------+------+------+------+------+------+
| TC,5 | EC,3 | C1,6 | C2,6 | C3,6 | C4,6 | C5,6 | C6,6 | C7,6 | C8,6 |
+------+------+------+------+------+------+------+------+------+------+

TC = 00100b = 4, and each symbol C1-C8 contains codes, that should match the indexes in this string:
#ABCDEFGHIJKLMNOPQRSTUVWXYZ#####_###############0123456789######

After decoding its easy to get the aircraft name: EWG7184

symbols = "#ABCDEFGHIJKLMNOPQRSTUVWXYZ#####_###############0123456789######"
code_str = ""
for p in range(8):
     c = int(bits_str[8 + 6*p:8 + 6*(p + 1)], 2)
     code_str += symbols[c]
print("Aircraft Identification:", code_str.replace('#', ''))

Airborne position

The name decoding was simple, but the coordinates are more complicated. They are transmitted in the form of 2 frames, even and odd. The field code TC = 01011b = 11.

Flightradar24 — how does it work? Part 2, ADS-B protocol - 11

Example of even and odd data frames:

01011 000 000101110110 00 10111000111001000 10000110101111001
01011 000 000110010000 01 10010011110000110 10000011110001000

The calculation of the coordinates itself uses a bit tricky formula:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 12
(source)

I am not a GIS expert, so I don't know where it comes from. Who knows it better, please write in comments.

The altitude calculation is simpler — depending on a specific bit, it can be represented as either a multiple of 25 or 100 feet.

Airborne Velocity

Dataframe with TC = 19. The interesting thing here is that the speed can be relative to the ground (more accurate, called Ground Speed), and Airspeed, measured by the aircraft air sensor (can be less accurate because of the wind). Many other different fields are also transmitted:

Flightradar24 — how does it work? Part 2, ADS-B protocol - 13
(source)

Conclusion

As we can see, ADS-B technology has become an interesting symbiosis, when a standard is widely usable not only for professionals, but also for ordinary users. But definitely, the key role in this was done by the cheapening of the of digital SDR receivers technology, which allows to receive signals with a frequency higher than gigahertz on a very cheap device.

The standard itself, of course, has much more data. Those interested can view the PDF on the page ICAO or visit the mode-s.org website already mentioned above. It is unlikely that this article will be really used by most of readers, but I hope, at least the general idea of how it works, is now more clear.

By the way, the ADS-B Python decoder already exists, it can investigated on the github. SDR receivers owners can also build and run ready-to-use ADS-B decoder from this page , and (I'll repeat again) some details we also in the first part of this article.

The parser source code, described above, is given under the spoiler. This is just a test example that does not pretend to production quality, but in general it works, and it can parse the WAV-file, recorded above.

Source Code (Python)

from __future__ import print_function

from scipy.io import wavfile
from scipy import signal
import matplotlib.pyplot as plt
import numpy as np
import math
import sys


def parse_message(data, start, bit_len):
    max_len = bit_len*128
    A = data[start:start + max_len]
    A = signal.resample(A, 10*max_len)
    bits = np.zeros(10*max_len)
    bit_len *= 10
    start_data = bit_len*8

    # Parse first 8 bits
    bits_str = ""
    for p in range(8):
        pos = start_data + bit_len*p
        p1, p2 = A[pos: pos + bit_len/2], A[pos + bit_len/2: pos + bit_len]
        avg1, avg2 = np.average(p1), np.average(p2)
        if avg1 < avg2:
            bits_str += "0"
        elif avg1 > avg2:
            bits_str += "1"

    df = int(bits_str[0:5], 2)

    # Aircraft address (db - https://junzis.com/adb/?q=3b1c5c )
    bits_str = ""
    for p in range(8, 32):
        pos = start_data + bit_len * p
        p1, p2 = A[pos: pos + bit_len / 2], A[pos + bit_len / 2: pos + bit_len]
        avg1, avg2 = np.average(p1), np.average(p2)
        if avg1 < avg2:
            bits_str += "0"
        elif avg1 > avg2:
            bits_str += "1"
    # print "Aircraft address:", bits_str, hex(int(bits_str, 2))
    address = hex(int(bits_str, 2))

    # Filter specific aircraft (optional)
    # if address != "0x3c5ee2":
    #    return

    if df == 16 or df == 17 or df == 18 or df == 19 or df == 20 or df == 21:
        # print "Pos:", start, "DF:", msg_type

        # Data (56bit)
        bits_str = ""
        for p in range(32, 88):
            pos = start_data + bit_len*p
            p1, p2 = A[pos: pos + bit_len/2], A[pos + bit_len/2: pos + bit_len]
            avg1, avg2 = np.average(p1), np.average(p2)
            if avg1 < avg2:
                bits_str += "0"
                # bits[pos + bit_len / 2] = 50
            elif avg1 > avg2:
                bits_str += "1"

        # http://www.lll.lu/~edward/edward/adsb/DecodingADSBposition.html
        # print "Data:"
        # print bits_str[:8], bits_str[8:20],  bits_str[20:22], bits_str[22:22+17], bits_str[39:39+17]
        # Type Code:
        tc, ec = int(bits_str[:5], 2), int(bits_str[5:8], 2)
        # print("DF:", df, "TC:", tc)
        
        # 1 - 4  Aircraft identification
        # 5 - 8  Surface position
        # 9 - 18  Airborne position (w/ Baro Altitude)
        # 19  Airborne velocities

        if tc >= 1 and tc <= 4: # and (df == 17 or df == 18):
            print("Aircraft address:", address)
            print("Data:")
            print(bits_str[:8], bits_str[8:14],  bits_str[14:20], bits_str[20:26], bits_str[26:32], bits_str[32:38], bits_str[38:44])

            symbols = "#ABCDEFGHIJKLMNOPQRSTUVWXYZ#####_###############0123456789######"
            code_str = ""
            for p in range(8):
                c = int(bits_str[8 + 6*p:8 + 6*(p + 1)], 2)
                code_str += symbols[c]
            print("Aircraft Identification:", code_str.replace('#', ''))
            print()
        if tc == 11:
            print("Aircraft address:", address)
            print("Data: (11)")
            print(bits_str[:8], bits_str[8:20],  bits_str[20:22], bits_str[22:22+17], bits_str[39:39+17])

            # Bit 22 contains the F flag which indicates which CPR format is used (odd or even)
            # First frame has F flag = 0 so is even and the second frame has F flag = 1 so odd
            # f = bits_str[21:22]
            # print("F:", int(f, 2))

            # Altitude
            alt1b = bits_str[8:20]
            if alt1b[-5] == '1':
                bits = alt1b[:-5] + alt1b[-4:]
                n = int(bits, 2)
                alt_ft = n*25 - 1000
                print("Alt (ft)", alt_ft)

            # lat_dec = int(bits_str[22:22+17], 2)
            # lon_dec = int(bits_str[39:39+17], 2)
            # print("Lat/Lon:", lat_dec, lon_dec)

            # http://airmetar.main.jp/radio/ADS-B%20Decoding%20Guide.pdf
            print()
        if tc == 19:
            print("Aircraft address:", address)
            print("Data:")
            # print(bits_str)
            print(bits_str[:5], bits_str[5:8], bits_str[8:10], bits_str[10:13], bits_str[13] ,bits_str[14:24], bits_str[24], bits_str[25:35], bits_str[35:36], bits_str[36:65])

            subtype = int(bits_str[5:8], 2)
            # https://mode-s.org/decode/adsb/airborne-velocity.html
            spd, hdg, rocd = -1, -1, -1
            if subtype == 1 or subtype == 2:
                print("Velocity Subtype 1: Ground speed")
            
                v_ew_sign = int(bits_str[13], 2)
                v_ew = int(bits_str[14:24], 2) - 1       # east-west velocity
                
                v_ns_sign = int(bits_str[24], 2)
                v_ns = int(bits_str[25:35], 2) - 1       # north-south velocity
                
                v_we = -1*v_ew if v_ew_sign else v_ew
                v_sn = -1*v_ns if v_ns_sign else v_ns
                
                spd = math.sqrt(v_sn*v_sn + v_we*v_we)  # unit in kts
                
                hdg = math.atan2(v_we, v_sn)
                hdg = math.degrees(hdg)                 # convert to degrees
                hdg = hdg if hdg >= 0 else hdg + 360    # no negative val
            if subtype == 3:
                print("Subtype Subtype 3: Airspeed")
                hdg = int(bits_str[14:24], 2)/1024.0*360.0
                spd = int(bits_str[25:35], 2)
            
            vr_sign = int(bits_str[36], 2)
            vr = int(bits_str[36:45], 2)
            rocd = -1*vr if vr_sign else vr         # rate of climb/descend
            print("Speed (kts):", spd, "Rate:", rocd, "Heading:", hdg)
            print()

        # print()

def calc_coordinates():
    def _cprN(lat, is_odd):
        nl = _cprNL(lat) - is_odd
        return nl if nl > 1 else 1

    def _cprNL(lat):
        try:
            nz = 15
            a = 1 - math.cos(math.pi / (2 * nz))
            b = math.cos(math.pi / 180.0 * abs(lat)) ** 2
            nl = 2 * math.pi / (math.acos(1 - a/b))
            return int(math.floor(nl))
        except:
            # happens when latitude is +/-90 degree
            return 1
    
    def floor_(x):
        return int(math.floor(x))
  
    lat1b, lon1b, alt1b = "10111000111010011", "10000110111111000", "000101111001"
    lat2b, lon2b, alt2b = "10010011101011100", "10000011000011011", "000101110111"
    lat1, lon1, alt1 = int(lat1b, 2), int(lon1b, 2), int(alt1b, 2)
    lat2, lon2, alt2 = int(lat2b, 2), int(lon2b, 2), int(alt2b, 2)
    
    # 131072 is 2^17, since CPR lat and lon are 17 bits each
    cprlat_even, cprlon_even = lat1/131072.0, lon1/131072.0
    cprlat_odd, cprlon_odd = lat2/131072.0, lon2/131072.0
    print(cprlat_even, cprlon_even)

    j = floor_(59*cprlat_even - 60*cprlat_odd)
    print(j)

    air_d_lat_even = 360.0 / 60
    air_d_lat_odd = 360.0 / 59

    # Lat
    lat_even = float(air_d_lat_even * (j % 60 + cprlat_even))
    lat_odd = float(air_d_lat_odd * (j % 59 + cprlat_odd))
    if lat_even >= 270:
        lat_even = lat_even - 360
    if lat_odd >= 270:
        lat_odd = lat_odd - 360

    # Lon
    ni = _cprN(lat_even, 0)
    m = floor_(cprlon_even * (_cprNL(lat_even)-1) - cprlon_odd * _cprNL(lat_even) + 0.5)
    lon = (360.0 / ni) * (m % ni + cprlon_even)
    print("Lat", lat_even, "Lon", lon)

    # Altitude
    # Q-bit (bit 48) indicates whether the altitude is encoded in multiples of 25 or 100 ft (0: 100 ft, 1: 25 ft)
    # The value can represent altitudes from -1000 to +50175 ft.
    if alt1b[-5] == '1':
        bits = alt1b[:-5] + alt1b[-4:]
        n = int(bits, 2)
        alt_ft = n*25 - 1000
        print("Alt (ft)", alt_ft)


fs, data = wavfile.read("adsb_20190311_191728Z_1090000kHz_RF.wav")
T = 1/fs

print("Sample rate %f MS/s" % (fs / 1e6))
print("Cnt samples %d" % len(data))
print("Duration: %f s" % (T * len(data)))

data = data.astype(float)

cnt = data.shape[0]
# Processing only part on file (faster):
# cnt = 10000000
# data = data[:cnt]
print("Processing I/Q...")
I, Q = data[:, 0], data[:, 1]
A = np.sqrt(I*I + Q*Q)

bits = np.zeros(cnt)

# To see scope without any processing, uncomment
# plt.plot(A)
# plt.show()
# sys.exit(0)

print("Extracting signals...")

pos = 0
avg = 200
msg_start = 0
# Find beginning of each signal
while pos < cnt - 16*1024:
    # P1 - message start
    while pos < cnt - 16*1024:
        if A[pos] < avg and A[pos+1] > avg and pos - msg_start > 1000:
            msg_start = pos
            bits[pos] = 100
            pos += 4
            break
        pos += 1

    start1, start2, start3, start4 = msg_start, 0, 0, 0
    # P2
    while pos < cnt - 16*1024:
        if A[pos] < avg and A[pos+1] > avg:
            start2 = pos
            bits[pos] = 90
            pos += 1
            break
        pos += 1
    # P3
    while pos < cnt - 16*1024:
        if A[pos] < avg and A[pos+1] > avg:
            start3 = pos
            bits[pos] = 80
            pos += 1
            break
        pos += 1
    # P4
    while pos < cnt - 16*1024:
        if A[pos] < avg and A[pos+1] > avg:
            start4 = pos
            bits[pos] = 70
            pos += 1
            break
        pos += 1


    sig_diff = start4 - start1
    if 20 < sig_diff < 25:
        bits[msg_start] = 500
        bit_len = int((start4 - start1) / 4.5)
        # print(pos, start1, start4, ' - ', bit_len)
        # start = start1 + 8*bit_len
        parse_message(A, msg_start, bit_len)

        pos += 450

# For debugging: check signal start
# plt.plot(A)
# plt.plot(bits)
# plt.show()

I hope it was useful, thanks for reading.

Автор: DmitrySpb79

Источник

* - обязательные к заполнению поля


https://ajax.googleapis.com/ajax/libs/jquery/3.4.1/jquery.min.js