這邊使用的 Container Host 為 CentOS 7.x，主要的要求是 Linux Kernel 需要 3.x 以上（可以用 CentOS 6.x、但是需要搭配 libvirt），並且採用 SystemD 作為 INIT 系統的 OS（方便後續的演練相容性）。
不使用 Docker 的理由：Docker Container 主在求精鍊，並打包成 Image 透過載入的方式產生運行的軟體服務。Docker Container 也傾向以 “1 image, 1 service” 來處理，不傾向在裡面啟動多個服務。Docker Container 裡面簡化 INIT 的支援，因此把 Docker Container 當成 OS 作演練比較麻煩。LXC Container 單純以 chroot 手法，以實體 OS 目錄運作，且完整支援 SystemD INIT，更加貼近實際架構設置環境，比起 Docker 長的更像一個 VM，比較適合作演練用途。
下圖為設置後的環境範例
這份筆記的範圍：
- 設置具有 LXC 的 VM 環境（Privileged Container）
- 設置基本 Container 網路，使 (1) Container 之間互通 (2) Container 可以訪問外面
這邊的 Container 使用的是 Privileged Container，使用 root 啟動、進入後也是 root 帳號，僅作為練習環境用。普通帳號用的 Unprivileged Container 不在這筆記的範圍。
Google Cloud Compute Engine 中的預備事項
Linux 中的網路設置，除了基本的 config 設置方式（/etc/sysconfig/network* 或是 /etc/rc.d/rc.network* 或是 /etc/init.d/network* 這類），大多有 NetworkManager 這個軟體進行第二層的控制，有時候會有一些預設值優先性的狀況。
在 Google 雲端的 VM 啟用的 CentOS 7.x VM，網路的設置有外部介入調控。後續 LXC 設定步驟會啟用內部的網路界面，也會受到 NetworkManager 的影響。因此，進行後續設置前，需要先將 NetworkManager 服務關閉並停用：
[user@lxc-lab ~]$ sudo chkconfig NetworkManager off [user@lxc-lab ~]$ sudo service NetworkManager stop
在普通電腦裝的單機 VM 軟體（VirtualBlx、VMWare、Qemu 等等），比較不會遇到這個狀況，且保留 NetworkManager 可以自動向 VM 程式取 IP，故本步驟在本機設定中可略。
設置 LXC 軟體
1) LXC 的軟體，在 EPEL Repo 裡面提供。在 CentOS 7.x 提供的是 LXC 1.x 版，LXC 2.x 才有提供資源分配（例如 CPU、RAM 之類的），不過這不影響測試環境的使用。
這邊同時補上 Container Host 需要的軟體
- Container 的 Bridge 網路需要的軟體跟檢測工具
- LXC 本身
- DNS 服務：以便使 Container 在連上網路之後可以認得網址
[user@lxc-lab ~]$ sudo yum install -y epel-release [user@lxc-lab ~]$ sudo yum install -y bridge-utils bind-utils net-tools telnet [user@lxc-lab ~]$ sudo yum install -y lxc lxc-templates lxc-extra [user@lxc-lab ~]$ sudo yum install -y dnsmasq [user@lxc-lab ~]$ rpm -qa | grep lxc lxc-libs-1.0.11-1.el7.x86_64 lxc-extra-1.0.11-1.el7.x86_64 python34-lxc-1.0.11-1.el7.x86_64 lua-lxc-1.0.11-1.el7.x86_64 lxc-1.0.11-1.el7.x86_64 lxc-templates-1.0.11-1.el7.x86_64 [user@lxc-lab ~]$
2) 安裝完之後，先依照需要調整 LXC v1 Container 初始化安裝的 Template 內容。
這邊補上常用的 less、net-tools、sudo、which 等指令，以及 httpd 服務。其中先補上 httpd 服務的原因是要避開 Container（LXC 跟 Docker）在不夠新的 Linux Kernel 共通的問題，因而需要事先安裝
[user@lxc-lab ~]$ ls /usr/share/lxc/templates/
lxc-alpine lxc-busybox lxc-debian lxc-gentoo lxc-oracle lxc-ubuntu
lxc-altlinux lxc-centos lxc-download lxc-openmandriva lxc-plamo lxc-ubuntu-cloud
lxc-archlinux lxc-cirros lxc-fedora lxc-opensuse lxc-sshd
[user@lxc-lab ~]$ sudo cp /usr/share/lxc/templates/lxc-centos /usr/share/lxc/templates/lxc-centos.orig
[user@lxc-lab ~]$
[user@lxc-lab ~]$ sudo vi /usr/share/lxc/templates/lxc-centos
[user@lxc-lab ~]$ ## 編輯之後
[user@lxc-lab ~]$ diff -Naur /usr/share/lxc/templates/lxc-centos.orig /usr/share/lxc/templates/lxc-centos
--- /usr/share/lxc/templates/lxc-centos.orig 2019-01-26 16:46:54.419687480 +0000
+++ /usr/share/lxc/templates/lxc-centos 2019-01-26 16:49:13.764836140 +0000
@@ -411,7 +411,7 @@
else
YUM="$YUM0"
fi
- PKG_LIST="yum initscripts passwd rsyslog vim-minimal openssh-server openssh-clients dhclient chkconfig rootfiles policycoreutils cronie"
+ PKG_LIST="yum initscripts passwd rsyslog vim-minimal openssh-server openssh-clients dhclient chkconfig rootfiles policycoreutils cronie less net-tools sudo which httpd"
# use temporary repository definition
# always prefer the repo given by the user
[user@lxc-lab ~]$
設置 LXC 環境的網路
1) 設定之前的網路狀況
[user@lxc-lab ~]$ ifconfig eth0: flags=4163mtu 1460 inet 10.128.0.5 netmask 255.255.255.255 broadcast 10.128.0.5 inet6 fe80::4001:aff:fe80:5 prefixlen 64 scopeid 0x20 ether 42:01:0a:80:00:05 txqueuelen 1000 (Ethernet) RX packets 6185 bytes 53637467 (51.1 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 5330 bytes 438492 (428.2 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [user@lxc-lab ~]$ [user@lxc-lab ~]$ brctl show bridge name bridge id STP enabled interfaces [user@lxc-lab ~]$ ip route default via 10.128.0.1 dev eth0 proto dhcp metric 100 10.128.0.1 dev eth0 proto dhcp scope link metric 100 10.128.0.5 dev eth0 proto kernel scope link src 10.128.0.5 metric 100 [user@lxc-lab ~]$ [user@lxc-lab ~]$ cat /etc/resolv.conf ; generated by /usr/sbin/dhclient-script search c.user-151709.internal. google.internal. nameserver 169.254.169.254 [user@lxc-lab ~]$
產生一個 Host 端 Bridge 網路界面
LXC 預設名稱叫做 virbr0，這邊會透過這個網卡界面，建立一個 Container 的子網段（NAT），這邊挑選 192.168.68.xxx 網段來使用，這邊使 Container Host 本身的 IP 為 192.168.68.10。該 IP 後續也是所有 Container 的 GATEWAY
[user@lxc-lab ~]$ cat /etc/lxc/default.conf lxc.network.type = veth lxc.network.link = virbr0 lxc.network.flags = up [user@lxc-lab ~]$ [user@lxc-lab ~]$ sudo tee /etc/sysconfig/network-scripts/ifcfg-virbr0 << EOF DEVICE="virbr0" BOOTPROTO="static" IPADDR="192.168.68.10" NETMASK="255.255.255.0" ONBOOT="yes" TYPE="Bridge" NM_CONTROLLED="no" EOF [user@lxc-lab ~]$ [user@lxc-lab ~]$ sudo service network restart Restarting network (via systemctl): [ OK ] [user@lxc-lab ~]$ [user@lxc-lab ~]$ ifconfig eth0: flags=4163mtu 1460 inet 10.128.0.5 netmask 255.255.255.255 broadcast 10.128.0.5 inet6 fe80::4001:aff:fe80:5 prefixlen 64 scopeid 0x20 ether 42:01:0a:80:00:05 txqueuelen 1000 (Ethernet) RX packets 9616 bytes 65538476 (62.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 8267 bytes 794066 (775.4 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73 mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 virbr0: flags=4163 mtu 1500 inet 192.168.68.10 netmask 255.255.255.0 broadcast 192.168.68.255 inet6 fe80::64a2:c6ff:fe54:2f9c prefixlen 64 scopeid 0x20 ether 66:a2:c6:54:2f:9c txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 14 bytes 908 (908.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 [user@lxc-lab ~]$ [user@lxc-lab ~]$ brctl show bridge name bridge id STP enabled interfaces virbr0 8000.000000000000 no [user@lxc-lab ~]$ [user@lxc-lab ~]$ ip route default via 10.128.0.1 dev eth0 10.128.0.1 dev eth0 scope link 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev virbr0 scope link metric 1003 192.168.68.0/24 dev virbr0 proto kernel scope link src 192.168.68.10 [user@lxc-lab ~]$
設置 IP Forwarding 的 Linux Kernel 參數
[user@lxc-lab ~]$ sudo tee /etc/sysctl.d/99-sysctl.conf << EOF
net.ipv4.ip_forward = 1
EOF
[user@lxc-lab ~]$
[user@lxc-lab ~]$ sudo sysctl -p
net.ipv4.ip_forward = 1
[user@lxc-lab ~]$ cat /proc/sys/net/ipv4/ip_forward
1
[user@lxc-lab ~]$
設置防火牆
內建防火牆在這邊不可以關閉，它負責進行封包轉發。以 virbr0 的封包，以 MASQUERADE 的方式轉往實際通往外部的網路界面 eth0
此外，這邊也需要把一些外部服務對 Container 打開，像是 DNS、SMTP Relay 等
這邊的設定，會以 XML 格式出現在 /etc/firewalld/ 裡面的 default.xml 以及 zones/internal.xml
[user@lxc-lab ~]$ sudo firewall-cmd --zone=internal --add-interface=virbr0 --permanent
success
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --zone=trusted --remove-interface=virbr0 --permanent
Warning: NOT_ENABLED: virbr0
success
[brandon_hsu@lxc-lab ~]$
[brandon_hsu@lxc-lab ~]$ sudo service firewalld restart
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING -o eth0 -j MASQUERADE -s 192.168.68.0/24
success
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i virbr0 -o eth0 -j ACCEPT
success
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 0 -i virbr0 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT
success
[brandon_hsu@lxc-lab ~]$
[brandon_hsu@lxc-lab ~]$ sudo cat /etc/firewalld/direct.xml
<?xml version="1.0" encoding="utf-8"?>
<direct>
<rule priority="0" table="filter" ipv="ipv4" chain="FORWARD">-i virbr0 -o eth0 -j ACCEPT</rule>
<rule priority="0" table="filter" ipv="ipv4" chain="FORWARD">-i virbr0 -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT</rule>
<passthrough ipv="ipv4">-t nat -I POSTROUTING -o eth0 -j MASQUERADE -s 192.168.68.0/24</passthrough>
</direct>
[brandon_hsu@lxc-lab ~]$
[brandon_hsu@lxc-lab ~]$ firewall-cmd --get-services
RH-Satellite-6 amanda-client amanda-k5-client bacula bacula-client bgp bitcoin bitcoin-rpc bitcoin-testnet bitcoin-testnet-rpc ceph ceph-mon cfengine condor-collector ctdb dhcp dhcpv6 dhcpv6-client dns docker-registry docker-swarm dropbox-lansync elasticsearch freeipa-ldap freeipa-ldaps freeipa-replication freeipa-trust ftp ganglia-client ganglia-master git gre high-availability http https imap imaps ipp ipp-client ipsec irc ircs iscsi-target jenkins kadmin kerberos kibana klogin kpasswd kprop kshell ldap ldaps libvirt libvirt-tls managesieve mdns minidlna mongodb mosh mountd ms-wbt mssql murmur mysql nfs nfs3 nmea-0183 nrpe ntp openvpn ovirt-imageio ovirt-storageconsole ovirt-vmconsole pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy proxy-dhcp ptp pulseaudio puppetmaster quassel radius redis rpc-bind rsh rsyncd samba samba-client sane sip sips smtp smtp-submission smtps snmp snmptrap spideroak-lansync squid ssh syncthing syncthing-gui synergy syslog syslog-tls telnet tftp tftp-client tinc tor-socks transmission-client upnp-client vdsm vnc-server wbem-https xmpp-bosh xmpp-client xmpp-local xmpp-server zabbix-agent zabbix-server
[brandon_hsu@lxc-lab ~]$
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --permanent --zone=internal --add-service=dns
success
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --permanent --zone=internal --add-service=smtp
success
[brandon_hsu@lxc-lab ~]$
[brandon_hsu@lxc-lab ~]$ sudo cat /etc/firewalld/zones/internal.xml
<?xml version="1.0" encoding="utf-8"?>
<zone>
<short>Internal</short>
<description>For use on internal networks. You mostly trust the other computers on the networks to not harm your computer. Only selected incoming connections are accepted.</description>
<interface name="virbr0"/>
<service name="ssh"/>
<service name="mdns"/>
<service name="samba-client"/>
<service name="dhcpv6-client"/>
<service name="dns"/>
<service name="smtp"/>
</zone>
[brandon_hsu@lxc-lab ~]$ sudo cat /etc/firewalld/zones/trusted.xml
<?xml version="1.0" encoding="utf-8"?>
<zone target="ACCEPT">
<short>Trusted</short>
<description>All network connections are accepted.</description>
</zone>
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --reload
success
[brandon_hsu@lxc-lab ~]$
[brandon_hsu@lxc-lab ~]$ sudo firewall-cmd --get-active-zones
internal
interfaces: virbr0
trusted
interfaces: eth0
[brandon_hsu@lxc-lab ~]$
設置 DNS 服務
[user@lxc-lab ~]$ sudo cp /etc/dnsmasq.conf /etc/dnsmasq.conf.orig [user@lxc-lab ~]$ sudo vi /etc/dnsmasq.conf [user@lxc-lab ~]$ diff -Naur /etc/dnsmasq.conf /etc/dnsmasq.conf.orig --- /etc/dnsmasq.conf 2019-01-26 17:38:09.436614224 +0000 +++ /etc/dnsmasq.conf.orig 2019-01-26 17:37:25.674803292 +0000 @@ -50,7 +50,7 @@ # to be up. Uncommenting this forces dnsmasq to try each query # with each server strictly in the order they appear in # /etc/resolv.conf -strict-order +#strict-order # If you don't want dnsmasq to read /etc/resolv.conf or any other # file, getting its servers from this file instead (see below), then @@ -108,7 +108,7 @@ #except-interface= # Or which to listen on by address (remember to include 127.0.0.1 if # you use this.) -listen-address=192.168.68.10 +#listen-address= # If you want dnsmasq to provide only DNS service on an interface, # configure it as shown above, and then use the following line to # disable DHCP and TFTP on it. [user@lxc-lab ~]$ [user@lxc-lab ~]$ sudo chkconfig dnsmasq on [user@lxc-lab ~]$ sudo service dnsmasq start
以上便完成環境的預備設置。接著就是下一步，直接操作 Container 了。請接著參考下一篇。
沒有留言:
張貼留言