Why all developers should learn how to perform basic network troubleshooting

Publish Date: Jun 6 '21

162 14

Regardless of whether you work on the front-end or back-end, I think all developers should gain some proficiency in network troubleshooting. This is especially true if you find yourself gravitating towards lower level systems programming.

The ability to troubleshoot the network and systems separates good developers from great developers. Great developers understand not just code abstraction, but understand the TCP/IP model:

Source: https://www.guru99.com/tcp-ip-model.html

Some basic network troubleshooting skills

If you are just getting into networking, here are some basic tools you should add to your toolbelt:

Perform a DNS query (e.g. dig or nslookup command)
Send an ICMP echo request to test end to end IP connectivity (i.e. ping command)
Analyze the various network hops (i.e. traceroute X.X.X.X)
Check whether you can establish a TCP socket connection (e.g. telnet X.X.X.X [port])
Test application layer (i.e. curl https://somedomain)
Perform a packet capture (e.g. tcpdump -i any) and what bits are sent on the wire

What IP address is my browser connecting to?

% dig dev.to

; <<>> DiG 9.10.6 <<>> dev.to
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39029
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;dev.to.                IN  A

;; ANSWER SECTION:
dev.to.         268 IN  A   151.101.2.217
dev.to.         268 IN  A   151.101.66.217
dev.to.         268 IN  A   151.101.130.217
dev.to.         268 IN  A   151.101.194.217

Is the web server listening on the HTTP port?

% telnet 151.101.2.217 443
Trying 151.101.2.217...
Connected to 151.101.2.217.
Escape character is '^]'.

Each of the above tools helps you isolate connectivity issues. For example, if your client receives an HTTP 5XX error, you can immediately rule out any TCP level issue. That is, you don't need to use telnet to check whether there's a firewall issue or whether the server is listening in on the right socket: the server already sent an application level response.

Summary

Learning more about the network stack helps you quickly pinpoint and isolate problems:

Is it my client-side application?
Is it a firewall blocking certain ports?
Is there a transient issue on the network?
Is the server up and running?

Let's chat more about network engineering and software development

If you are curious about learning how to move from front-end to back-end development, or from back-end development to low level systems programming, hit me up on Twitter: @memattchung

Comments 14 total

Simon FosterJun 6, 2021
As a developer that started his career as a sysadmin I agree with this article. Having a basic understanding of DNS and other network stuff has certainly been very useful to me.
- memattchungJun 6, 2021
  Hey Simon,
  
  I had a similar career trajectory, also starting off with systems administration. It definitely goes both ways: great sysadmins know some elements of programming (e.g. using a debugger, analyzing a core dump) and great programmers know some systems/network administration (e.g. checking for host-to-host connectivity, inspecting packet payloads). In this context, the 80/20 rule — 20% of activities contribute to 80% of results — rings true.
  
  These days, what are you working on?
Niitn KatageriJun 7, 2021
Completely agree, developers not having a clue about how different components of the system talk to eachother is really sad.

During college days I had inclination towards system admin but ended up on the programming side of things, but my inclination has helped me triage issues in development much faster.

Recently got into tcpdumps and port monitoring to analyse why a webservice request to one of engines was randomly erroring with connection reset.
- memattchungJun 7, 2021
  Agree. The best developers I've worked with weren't just great at programming, but also intimately aware of the underlying system too.
  
  webservice request to one of engines was randomly erroring with connection reset
  
  Another great tool you might want to use is strace, assuming your application is running on linux.
  - Niitn KatageriJun 8, 2021
    Sure will try it out. Thanks. Yes it runs on Linux.
Jonathan BoudreauJun 7, 2021
Personally I prefer tcpflow over tcpdump since it provides colouring which can make things easier to view. This is especially useful on systems where you can't easily transfer files (ie, the packet capture file) from.

In my experience ICMP is usually blocked. nmap can instead be used to test connectivity on the relevant port with a syn scan.
- memattchungJun 7, 2021
  Cool, I never heard of tcpflow. However, it appears this tool only handles tcp, not other transport protocols like udp, right?
  
  Also, regarding icmp, although it is often blocked, you can modify the encapsulation so instead of ip->icmp, you can modify it such that it rides on udp.
  - Jonathan BoudreauJun 7, 2021
    
    Cool, I never heard of tcpflow. However, it appears this tool only handles tcp, not other transport protocols like udp, right?
    
    Yea, in that case you should use tcpdump or something else. I just like the colouring.
    
    Also, regarding icmp, although it is often blocked, you can modify the encapsulation so instead of ip->icmp, you can modify it such that it rides on udp.
    
    Only what is absolutely necessary is let through by the firewall in my past experience. This includes disabling inbound UDP if the application does not need it. SYN scans work because you can poke the ports being used by the application.
- Niitn KatageriJun 8, 2021
  Thanks Jonathan, will use tcpflow next time I deal with the packets and the directions.
Jonas JohnsonJun 10, 2021
This is so true, I just ran into this on a project I was working on and didn’t realize my isp was auto resolving sites for me when the weren’t giving me all the wrong results. Thanks for sharing this!
- memattchungJun 10, 2021
  Interesting ! Can you elaborate on this — was your DNS pointing at your ISP's resolver?
  - Jonas JohnsonJun 11, 2021
    I searched for something and the dns responded, even though the site didn’t exist. I then called the non existent thing and it came back with a 200 response. You can probably tell I’m very new to this and so I can’t elaborate much. My mentor caught it because it just didn’t make any sense.
    - memattchungJun 11, 2021
      I think I have a better understanding. Likely the DNS record resolved returned an IP address pointed at a different address. Despite hitting that IP and receiving an HTTP 200 OK, the web site does not exist. Well, that's my guess at least.
      - Jonas JohnsonJun 11, 2021
        Yeah that sounds about right.

Add comment

memattchung @memattchung