apache2 and sshd are listening, but only sshd is accessible from remote

I've just installed apache2 on Ubuntu (aws) using chef.

Apache2 on Ubuntu Server: 10.1.1.1

Client: 10.1.1.200

I've also verified that both tcp 22 (sshd) & tcp 80 (apache2) are listening

root@Ubuntu:/var/www# netstat -anp | egrep '22|80'
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1469/sshd
tcp6 0 0 :::80 :::* LISTEN 16950/apache2
tcp6 0 0 :::22 :::* LISTEN 1469/sshd
root@Ubuntu:/var/www# 

Processes also look good

root@Ubuntu:/var/www# ps -ef | egrep 'apache2|sshd'
root 1469 1 0 Sep29 ? 00:00:00 /usr/sbin/sshd -D
root 16950 1 0 Sep29 ? 00:00:00 /usr/sbin/apache2 -k start
www-data 16953 16950 0 Sep29 ? 00:00:02 /usr/sbin/apache2 -k start
www-data 16954 16950 0 Sep29 ? 00:00:02 /usr/sbin/apache2 -k start
root@Ubuntu:/var/www# 

Service also running as usual

root@Ubuntu:/var/www# service apache2 status
● apache2.service - LSB: Apache2 web server Loaded: loaded (/etc/init.d/apache2; bad; vendor preset: enabled) Drop-In: /lib/systemd/system/apache2.service.d └─apache2-systemd.conf Active: active (running) since Fri 2017-09-29 23:36:49 UTC; 2h 13min ago Docs: man:systemd-sysv-generator(8) Tasks: 55 Memory: 6.5M CPU: 4.637s CGroup: / ├─16950 /usr/sbin/apache2 -k start ├─16953 /usr/sbin/apache2 -k start └─16954 /usr/sbin/apache2 -k start
Sep 29 23:36:47 Ubuntu systemd[1]: Starting LSB: Apache2 web server...
Sep 29 23:36:47 Ubuntu apache2[16933]: * Starting Apache httpd web server apache2
Sep 29 23:36:48 Ubuntu apache2[16933]: AH00557: apache2: apr_sockaddr_info_get() failed for Ubuntu
Sep 29 23:36:48 Ubuntu apache2[16933]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the '
Sep 29 23:36:49 Ubuntu apache2[16933]: *
Sep 29 23:36:49 Ubuntu systemd[1]: Started LSB: Apache2 web server.
root@Ubuntu:/var/www# 

However, when I try to access the server remotely via , nothing happen. I don't have this problem with ssh on the same server.

I've also run tcpdump when trying to access the web server from remote but I did not see anything

Test from client

[root@Client]# curl 10.1.1.1:80
curl: (7) Failed connect to 10.1.1.1:80; Connection timed out
[root@Client]# 

But nothing seen on the server

root@Ubuntu:/var/www# tcpdump -nni eth0 port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
root@Ubuntu:/var/www# 

However, when I try the same thing with ssh, I'm able to access it remotely and traffic seen on tcpdump as expected.

[root@Client]# curl 10.1.1.1:22
SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
curl: (56) Recv failure: Connection reset by peer
[root@Client]# 

tcpdump on ssh

root@Ubuntu:/var/www# tcpdump -nni eth0 port 22 and host 10.1.1.200
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
02:03:04.320761 IP 10.1.1.200.35890 > 10.1.1.1.22: Flags [S], seq 1658944714, win 26883, options [mss 8961,sackOK,TS val 52461068 ecr 0,nop,wscale 7], length 0
02:03:04.320923 IP 10.1.1.1.22 > 10.1.1.200.35890: Flags [S.], seq 2270168630, ack 1658944715, win 26847, options [mss 8961,sackOK,TS val 13211819 ecr 52461068,nop,wscale 7], length 0
02:03:04.321264 IP 10.1.1.200.35890 > 10.1.1.1.22: Flags [.], ack 1, win 211, options [nop,nop,TS val 52461069 ecr 13211819], length 0
02:03:04.321287 IP 10.1.1.200.35890 > 10.1.1.1.22: Flags [P.], seq 1:80, ack 1, win 211, options [nop,nop,TS val 52461069 ecr 13211819], length 79
02:03:04.321293 IP 10.1.1.1.22 > 10.1.1.200.35890: Flags [.], ack 80, win 210, options [nop,nop,TS val 13211819 ecr 52461069], length 0
02:03:04.326160 IP 10.1.1.1.22 > 10.1.1.200.35890: Flags [P.], seq 1:42, ack 80, win 210, options [nop,nop,TS val 13211821 ecr 52461069], length 41
02:03:04.326683 IP 10.1.1.1.22 > 10.1.1.200.35890: Flags [R.], seq 42, ack 80, win 210, options [nop,nop,TS val 13211821 ecr 52461069], length 0
02:03:04.326899 IP 10.1.1.200.35890 > 10.1.1.1.22: Flags [.], ack 42, win 211, options [nop,nop,TS val 52461075 ecr 13211821], length 0
02:03:04.326911 IP 10.1.1.1.22 > 10.1.1.200.35890: Flags [R], seq 2270168672, win 0, length 0
^C
9 packets captured
9 packets received by filter
0 packets dropped by kernel
root@Ubuntu:/var/www# 

Update: iptables output

Also, this box is installed on aws

root@Ubuntu:~# iptables -vL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination
root@Ubuntu:~# 

2 Answers

Apache is listening on IPv6, not IPv4:

tcp6 0 0 :::80 :::* LISTEN 16950/apache2

tcp6 and :::80 are both clues that Apache is listening on IPv6, not v4 (these would be tcp and 0.0.0.0:80 otherwise).

There are many solutions to this problem. Probably the best one is to tell Apache to explicitly listen on IPv4. Change your Listen directive from something like this:

Listen 80

to:

Listen 0.0.0.0:80

Thanks guys for your feedback. Apparently, aws only open ssh (tcp22) for this connectivity.

Once http (tcp80) is permitted on aws, I can access this box remotely.

I knew something was blocking as nmap shows filtered on tcp80 and open on tcp22, but had no clue it was aws at that moment.

Before

Nmap scan report for 10.1.1.1
Host is up (0.062s latency).
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 7.2p2 Ubuntu 4ubuntu2.2 (Ubuntu Linux; protocol 2.0)
80/tcp filtered http nginx 1.10.3 (Ubuntu)

After

Nmap scan report for 10.1.1.1
Host is up (0.062s latency).
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 7.2p2 Ubuntu 4ubuntu2.2 (Ubuntu Linux; protocol 2.0)
80/tcp open http nginx 1.10.3 (Ubuntu)