Linux's commands and tricks I'm using in my daily job as a developer
Mateusz Jarzyna

Mateusz Jarzyna @mateuszjarzyna

Location:
Poland
Joined:
May 9, 2019

Linux's commands and tricks I'm using in my daily job as a developer

Publish Date: Nov 11 '19
823 45

This is not a post from the series of those describing the cd command. It's just a list of commands and tricks I'm using (almost) every day.

Port forwarding

Sometimes I have to connect to database and of course I prefer to use my GUI manager (JetBrains DataGrip).
So, if security policy exist in your company and your database's port is not exposed you can execute

ssh -L{port on your PC}:localhost:{database's port} root@{server IP}

The command below will open port 3308 on your laptop and everything will be forwarded to 192.168.1.2:3306

ssh -L3308:localhost:3306 root@192.168.1.2

localhost means that database is listening on 192.168.1.2. You can type for example 192.168.3.77 and everything will be forwarded to .3.77 server via .1.2.

Edit file in VIM without sudo, but save with sudo

Have you ever edited some configs file and forgot to sudo? Me too... There is a trick to save the file anyway, just type in VIM:

:w !sudo tee %

Explanation

Go to beggining/end of line in terminal

If you wrote a very long command in the terminal it may take a long time before you return to the begging of the line to add missing sudo. And back to the end to add some parameters.
Press crtl + a to move to the begging and crtl + e to the end of the line in terminal.

ll

Save few days in a year by typing ll instead of ls -la. Works on most Linux servers.

Execute command you executed in the past

Last command

To execute last command over again you can of course press ↑ (arrow up) key. But you can also type !!. So executing last command as a root is very easy

sudo !!

To run the last command that started with apt type !apt

Search history

To find the command that contains /tmp you have executed in the past press ctrl + r and type /tmp. Press ctrl + r again for next result.
To show all commands or to search using regular expression use

history | grep "/tmp"

Agree for everything

To say yes for each question you can use application called yes

yes | yum install curl

use yes no to say no and discard.


WARNING

As @patricnox notices in the comment - using yes may do unexpected things. You can accidentally install 10 GB of dependencies or other things you don't want to do.


Run a long-lasting process in the background and close the terminal

If you run a script that will end in 3 days, you don't have to wait with the terminal window open to end. You can run it using nohup command

nohup wget http://large-files.com/10gb-super-movie.avi &

wget works in the background, output is saved to nohup.out file in working directory.

Checking who has stolen your favourite port

It's really annoying when you are trying to run nginx but you can't because there is already apache running and port 443 is busy.
So, how to determinate which process is listening on port 80:

$ netstat -tulpn | grep 80
tcp6       0      0 :::80                 :::*                   LISTEN     10177/java

10177 is a pid you are looking for. Now execute

ps aux | grep 10177

for more details.

Reading logs

Everyone knows that less is a very good way to read a logs files. But you can also read gziped logs without extracting!

less /var/log/my-app/my-app.log.2015.12.14.gz

Live reading

tail -f /var/log/my-app/my-app.log | grep ERROR

The command above will show only new lines that contains ERROR.

Sort process

Show top 3 processes sorted by CPU usage

ps aux --sort=-pcpu | head -n 4

Show top 3 processes sorted by memory usage

ps aux --sort=-rss | head -n 4

Executing command every X seconds

To print command's output every X seconds you can use watch command. For example to create clock run

watch -n 1 date

Quiet mode

A lot of standards commands has quiet or silent mode. Very useful when you are creating some bash script. In most of the cases just add -q or -s (read --help or man or check on StackOverflow)

zip -q archive.zip big-file.jpg

But sometimes (practically always with in-house scripts) you have to ignore the output (send to /dev/null)

./very-verbose.sh 1>/dev/null

Create log files for scripts executed by crontab

0 22 * * 1-5 /opt/scripts/send-report.sh 2>/var/log/scripts/report-error.log

So next time when your script will fail you won't lose the reason

Comments 45 total

  • PatricNox
    PatricNoxNov 11, 2019

    Working with drush, the last one is very neat!

    I have a concern regarding "Yes". I find it a bad idea to use this since what if you encounter a new tool? Maybe you install something and Yes makes it install, or maybe skip, dependencies that's crucial to have/not have in the project

    (Vague example scenario, but you get the point!)

    • Mateusz Jarzyna
      Mateusz JarzynaNov 11, 2019

      Yup, you are right. yes program may do unexpected thing, but sometimes you do know the script very well and you know all the questions.
      Maybe I should change the example and add some warning

    • Mateusz Jarzyna
      Mateusz JarzynaNov 11, 2019

      Thanks, PatricNox! I've added little warning in the post

    • Ben Sinclair
      Ben SinclairNov 11, 2019

      A lot of commands take a -y flag too.

      If you're particularly concerned about it doing something you don't want, you can use an expect script instead. It's kind of like selenium for the command line.

      • PatricNox
        PatricNoxNov 11, 2019

        Yep! I tend to use the y flag.

  • Dinesh Pandiyan
    Dinesh PandiyanNov 11, 2019

    Magic, right in your command line! ✨

    Thanks for sharing these.

  • Modaf
    ModafNov 11, 2019

    Omfg the yes command. Love that one

  • yeedle
    yeedleNov 11, 2019

    If you're using port forwarding for DayaGrip, you can use the connection settings dialog to set it directly (through the ssh tab).

    • Mateusz Jarzyna
      Mateusz JarzynaNov 11, 2019

      Lol, you're right, I've missed that feature. Anyway it's still useful for Kibana or other services that are not exposed publicly.

  • Nick Trierweiler
    Nick TrierweilerNov 11, 2019

    You have a typo:
    "jut type in VIM" should say "just type in VIM" :)

  • Fred Richards
    Fred RichardsNov 11, 2019

    I'm a huge fan of one-liners and simple hacks, these are great! I'm going to start using the vim and sudo one immediately.

  • Vinay Hegde
    Vinay HegdeNov 11, 2019

    If you've a common SSH login with sudo (shouldn't be the case mostly), do be careful with !! as you could end up running a potentially dangerous command in a hurry.

    You could use sudo lsof -i :80 to identify which app is taking up a port instead of netstat & ps

    Also zless to view gzipped logs without extraction & wget -c URL to retry downloads in case of issues.

  • mgfrobzz
    mgfrobzzNov 12, 2019

    If you have to telecommute, and your company's vpn connection is sh*tty, and they don't allow you to ssh in, reverse ssh tunneling is your friend:
    thegeekstuff.com/2013/11/reverse-s...

    • Jack Kingsman
      Jack KingsmanNov 12, 2019

      Also a good way to get your security team grumpy with you ;) Outbound SSH from prod sets off about a dozen alarm bells for us. Talking to your manager or DevOps team member is another option for addressing rough inbound access :)

  • Charles Banas
    Charles BanasNov 12, 2019

    If you're editing config files with vim and need to use that sudo tee hack, stop.

    You should be using sudoedit. It safely copies the config to /tmp and runs $EDITOR as your regular user with your user config, and only overwrites the file you're editing if you actually save. This is far more desirable than running your editor as root.

    • Maxime Moreau
      Maxime MoreauNov 12, 2019

      Very good point, I think that we should never use sudo vim.

  • simonced
    simoncedNov 12, 2019

    Those are great tips.

    In the terminal, one trick I like is going back to previous folder with

    cd -
    

    (that is a minus sign)

    Say, you are in /var/logs and you cd /data/dev/myproject, then you can go back to logs with cd -.

    Useful in some situations only, but quite handy nonetheless.

  • Utkarsh Talwar
    Utkarsh TalwarNov 12, 2019

    Great post, Mateusz! I shared it in our Telegram newsletter/channel for devs. 👉🏼 Link

  • mochsner
    mochsnerNov 12, 2019

    Found this extremely helpful! Thank you

  • iamnielsjanssen
    iamnielsjanssenNov 12, 2019

    For me 'screen' is unmissable when working in terminal applications. It is like having multiple windows! ctrl-a then 'c' creates new screen, ctrl-a 'space' switches through screens.

    • Keith Sloan
      Keith SloanNov 15, 2019

      Agreed. I tend to map my F-keys to swich between next, prev and list screens.
      bindkey -k k3 prev
      bindkey -k k4 next
      bindkey -k k5 windowlist

    • Jon Austin
      Jon AustinNov 20, 2019

      generally use tmux nowadays

  • rdilare
    rdilareNov 12, 2019

    A really useful post, thanks for sharing these. Specially ctrl+a and ctrl+e trick.

    I can relate to the missing sudo example😁😁😁

  • henry kirya
    henry kiryaNov 13, 2019

    I also use the home and end buttons to navigate to the start or ending of a long terminal command

  • RAJENDRASINH PARMAR
    RAJENDRASINH PARMARNov 13, 2019

    Nice and useful.

  • SerjP
    SerjPNov 13, 2019

    A "nice" command with 'yes' is:

    yes >/dev/sdaX &

    where X is 1, 2, etc
    Wipes your hdd with "yes".
    :)))

  • Patrick Hanford
    Patrick HanfordNov 13, 2019

    Awesome list.

    I vote to add ctrl + r for searching your previous commands to this list!

    Testing Gunicorn or Prometheus or something with a long pipeline? Run it once then ctrl + r and type some keywords to find it again without having to type it all out or mash the up arrow.

  • Sergio Gragera
    Sergio GrageraNov 13, 2019


    rsync -avz --delete --progress root@{server IP}:/dir_from_sync /dir_to_sync

  • Thomas H Jones II
    Thomas H Jones IINov 13, 2019

    Last command
    To execute last command over again you can of course press ↑ (arrow up) key. But you can also type !!. So executing last command as a root is very easy

    While !! is great, the overall ! method becomes really powerful if you understand that you're neither limited to re-executing just the last command or re-executing the command relatively unmodified.

    • Previously executed ssh host1.my.domain and now need to connect to host2.my.domain? Execute either of ^host1^host2^ or !!:s/host1/host2
    • Related to the latter, if you execute history and see that you previously connected to host1 with the 23rd command in your history-buffer, you could do !23:s/host1/host2
    • Have a command in your history that had a repeating string that you'd like to substitute all values for? !43:gs/orig_string/new_string

    Agree for everything

    I would be suuuuuuuper leery about developing habits around the yes command. Great sadness can ensue from habituating to its use ...especially if you have privileged access to a system. For commands that implement a built-in auto-yes feature (e.g. your yum does via the -y flag). Even there, I'd tend to avoid auto-yes except in the very specific context of scripted routines where you've validated the the auto-acknowleged behavior always acts the way you expect and need it to.

    It's also worth noting that not every command you use will understand accepting a yes from piped <STDIN>

    Run a long-lasting process in the background and close the terminal

    There are a non-trivial number of commands that don't react well to being backgrounded. You might think, "lemme background this thing and let it go about its business" only to come back minutes/hours/days later to find that it's done nothing. Basically, whenever you background a command, it's always a good idea to run the jobs command immediately afterwards to verify that it's in a running state. You may, instead, find that your backgrounded command is in a stopped state.

    Even better than using shells' built-in job-control for long-running tasks may be to use a terminal-multiplexer like screen or tmux. They're also great if you're connecting to a system over crappy links (remote in, fire up screen, kick off your tasks ...even if your connection dies, stuff keeps running and you can re-connect and re-attach to your session to finish things up).

  • Vladimir Nikolic
    Vladimir NikolicNov 14, 2019

    Your title is misleading kind of.
    Says commands you use in daily job, implies you are using them every day? Or that you use those command in your daily job as developer. If second whats your night job?
    I dont see any command i would run daily.
    Sorry to say but i dont see any of these commands so useful for daily work.
    Hard to believe you would be executing ssh -L{port on your PC}:localhost:{database's port} root@{server IP} daily (or almost) instead of creating an alias, same as for the other longer command you mention.
    Or this:
    0 22 * * 1-5 /opt/scripts/send-report.sh 2>/var/log/scripts/report-error.log
    Really? Almost daily?
    Cant take this seriously :)

  • Patryk
    PatrykNov 16, 2019

    alt + . is a great one too, in bash.

    $ mkdir -p /tmp/some/nested/directory
    $ cd <alt+.> # types the LAST argument - /tmp/some/nested/directory

    Re: netstat, it still works, of course, and you can continue using it, but it is deprecated in favour of ss.

    Re: sshing into a DB server as root, no offence, but I don't believe your company actually has a security policy :D (hopefully it was just an example).

    • Mateusz Jarzyna
      Mateusz JarzynaNov 16, 2019

      Re: sshing into a DB server as root, no offence, but I don't believe your company actually has a security policy :D (hopefully it was just an example).

      Yup, it was a 'funny' example

    • Guillermo Tobar
      Guillermo TobarNov 19, 2019

      For years I looked for this, and finally found it. Thank you very much for your contribution Patryk.

  • the-harry
    the-harryNov 19, 2019

    Great tips! I think it worth to mention the !$ operator in addition to !!.
    Where !$ is the argument of last command.
    So if you:

    mkdir foo

    You can enter the folder typing:

    cd !$

  • Tim Osborn
    Tim OsbornNov 19, 2019

    Follow mode for less! less +F

  • Sreeni
    SreeniNov 19, 2019

    CTRL - R is best to scroll through history

  • Araslanov Eugene
    Araslanov EugeneNov 19, 2019

    This also works in git

      git checkout -
    
    • Luan Fonseca
      Luan FonsecaNov 19, 2019

      A lot of git commands works with the -

      Like:

      (my-branch*) $ git checkout develop
      Switched to branch 'develop'
      Your branch is up-to-date with 'origin/develop'.
      
      (develop*) $ git checkout my-branch
      Switched to branch 'my-branch'
      
      (my-branch*) $ git merge -
      Already up-to-date.
      
  • Gergely Polonkai
    Gergely PolonkaiNov 19, 2019

    ll is actually a convenient alias set on most systems. Aliases are really powerful in and of themselves and worth a separate article (in fact, there’s a lot out there…)

  • James Miranda
    James MirandaNov 19, 2019

    Great tips!

  • Rafael Corrêa Gomes
    Rafael Corrêa GomesNov 20, 2019

    Excellent, my favorite was the !!, thank you for sharing it!

  • Abhishek Shingane
    Abhishek ShinganeNov 22, 2019

    Do you have to do compressing and uncompressing of files/folders?

    Check disk usage?

    List files inside a zip?

Add comment