Docker Compose in Windows System

[Saikiran Garapati]

Hi Team,

As per the below documentation link, Is it mandatory to have a Linux box to run the ./installSVDocker.sh or is there anywau we can run that file in windows 10 system.

I have installed docker in my Windows 10.0 system and able to access the ca devtest docker repository sv-docker.packages.ca.com and able to pull the images and when i run docker run commands i am able to see containerIds are getting listed 

docker run sv-docker.packages.ca.com/sv/iaam

docker run sv-docker.packages.ca.com/sv/config-server

docker run sv-docker.packages.ca.com/sv/lisa

docker run sv-docker.packages.ca.com/sv/portal

docker run sv-docker.packages.ca.com/sv/virtual-service-catalog

docker ps –a

ContainerID	IMAGE	COMMAND	CREATED	STATUS	PORTS	NAMES
6e8faed5bcb6	sv-docker.packages.ca.com/sv/virtual-service-catalog	"/bin/sh -c ./docker…"	3 hours ago	Up 3 hours	8080/tcp	festive_hodgkin
f7613ef69437	sv-docker.packages.ca.com/sv/portal	"/bin/sh -c ./docker…"	3 hours ago	Up 3 hours	1507/tcp	beautiful_shter
da9a39ed41d3	sv-docker.packages.ca.com/sv/lisa	"sh /devtest/docker-…"	3 hours ago	Up 3 hours	1505/tcp, 1528/tcp, 2009-2011/tcp, 2013-2014/tcp	objective_brahmagupta
dcd3ed841f65	sv-docker.packages.ca.com/sv/config-server	"java -Djava.securit…"	3 hours ago	Up 3 hours	8888/tcp	affectionate_williams
7b2a4cdde314	sv-docker.packages.ca.com/sv/iaam	"sh docker-entrypoin…"	3 hours ago	Up 3 hours		naughty_mclean

How do I access any component by above container ids and ports,
and this repo da9a39ed41d3sv-docker.packages.ca.com/sv/lisa is having 4 containers i believe why only one container id is showing?

DevTest with Docker Compose

Broadcom		remove preview
DevTest with Docker Compose You can deploy all DevTest containers in a single node using Docker-Compose. You can use a single command to create and start all services, and the options in the provided script let you deploy DevTest Solutions in a variety of ways, such as with multiple coordinators and VSEs. View this on Broadcom >

https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/devops/devtest-solutions/10-6/installing/DevTest-with-Docker-Compose.html

[Koustubh Warty]

@Giridharan Jagadeesan - please assist. ​

------------------------------
Head of Service Virtualization Product Management
CA Technologies - A Broadcom Company
------------------------------

[Saikiran Garapati]

Thanks Koustubh. 

@Giridharan Jagadeesan  Could you please give your suggestions on this.

[Giridharan Jagadeesan]

Hi Saikiran,

Thanks for your query. Docker compose for Windows 10 is not supported OOTB. Yes, Linux distribution is mandatory to run ./installSVDocker.sh. Today, the installSVDocker.sh is confined for Linux distros (Ubuntu, CentOS & RHEL) and in order to support Windows, the script will have to be enhanced. I have added this to our Backlog for Windows support.

And lisa image is a package that includes Registry, VSE, Coordinator and Simulator (by default Registry will start) and hence you will have only one container id.

Cheers
Giri

[Saikiran Garapati]

Hi Jagadessan, 

Thank you very much for the reply. We will try to use Ubuntu, CentOS & RHEL Os's fro Docker compose.

[Saikiran Garapati]

Hi Jagadessan,

We are using Linux box and trying to install ./installSVDocker.sh this file. but since we do not have Nw connectivity (No Internet Connection), we are getting the below error.
Is there any way we can use the local images instead of fetching them from their registry.

Please also let me know once we pull the images from repository where those images stored locally. I have pulled the images in windows system and did not see anywhere stored in my system?

sudo ./installSVDocker.sh -d https://XXXXXXXXXXXXXXXXXX:1506

Prerequisite checks

Checking for Docker

        Docker is already installed

        Docker version is 19.03

Check for Docker-Compose

        docker-compose is already installed

        docker-compose version 1.26.0, build d4451659

Checks if Enterprise Dashboard is accessible and other prerequisite checks

The provided enterprise dashboard https://XXXXXXXXXXXXXXXX:1506 is up.

Creating User and Group with name devtest and id 1010 if not present

Useful Information regarding VSEs

        Virtual Services should start in below range for Virtual Service Environment - vse

        20000-20049

Installation Directory is:

Pulling Images

Creating lisa-network and starting containers

Creating network "lisa-network" with driver "bridge"

Pulling config-server (sv-docker.packages.ca.com/sv/config-server:0.0.2)...

ERROR: Get https://sv-docker.packages.ca.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

telnet sv-docker.packages.ca.com 8443

Trying 75.126.0.63...

telnet: connect to address 75.126.0.63: Connection timed out

-------------------------------------------------------------------------
The below content is in Windows system:

docker ps –a

ContainerID	IMAGE	COMMAND	CREATED	STATUS	PORTS	NAMES
6e8faed5bcb6	sv-docker.packages.ca.com/sv/virtual-service-catalog	"/bin/sh -c ./docker…"	3 hours ago	Up 3 hours	8080/tcp	festive_hodgkin
f7613ef69437	sv-docker.packages.ca.com/sv/portal	"/bin/sh -c ./docker…"	3 hours ago	Up 3 hours	1507/tcp	beautiful_shter
da9a39ed41d3	sv-docker.packages.ca.com/sv/lisa	"sh /devtest/docker-…"	3 hours ago	Up 3 hours	1505/tcp, 1528/tcp, 2009-2011/tcp, 2013-2014/tcp	objective_brahmagupta
dcd3ed841f65	sv-docker.packages.ca.com/sv/config-server	"java -Djava.securit…"	3 hours ago	Up 3 hours	8888/tcp	affectionate_williams
7b2a4cdde314	sv-docker.packages.ca.com/sv/iaam	"sh docker-entrypoin…"	3 hours ago	Up 3 hours		naughty_mclean

How do I access any component by above container ids and ports. Is it like tcp://beautiful_shter:1507, if i give that can i get the access to portal? Please confirm me?

[Giridharan Jagadeesan]

Hi Saikiran,

Are you saying, you have pulled the images in Windows and trying to run the installSVDocker.sh from Linux without internet connection? If yes, this cross platform combination will not work. But try with network connectivity ensuring both your Windows and Linux machines are within same network. I suggest you to pls use only a Linux distro and follow the documentation - https://techdocs.broadcom.com/content/broadcom/techdocs/us/en/ca-enterprise-software/devops/devtest-solutions/10-5/installing/DevTest-with-Docker-Compose.html#concept.dita_0083fd89-5134-4ff0-b1aa-bdefe1c88ac0

And for e.g. to launch Portal - http://<hostname>:<port>/devtest (once you have everything up and running in a Linux machine, accessing components is similar to how you do in on-prem).

Cheers,
Giri

[Saikiran Garapati]

Hi Jagadessan,

No we are trying in Linux box only but that Linux box does not have the internet connection.

Hence we are getting the below error:

sudo ./installSVDocker.sh -d https://XXXXXXXXXXXXXXXXXX:1506

Prerequisite checks

Checking for Docker

        Docker is already installed

        Docker version is 19.03

Check for Docker-Compose

        docker-compose is already installed

        docker-compose version 1.26.0, build d4451659

Checks if Enterprise Dashboard is accessible and other prerequisite checks

The provided enterprise dashboard https://XXXXXXXXXXXXXXXX:1506 is up.

Creating User and Group with name devtest and id 1010 if not present

Useful Information regarding VSEs

        Virtual Services should start in below range for Virtual Service Environment - vse

        20000-20049

Installation Directory is:

Pulling Images

Creating lisa-network and starting containers

Creating network "lisa-network" with driver "bridge"

Pulling config-server (sv-docker.packages.ca.com/sv/config-server:0.0.2)...

ERROR: Get https://sv-docker.packages.ca.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

telnet sv-docker.packages.ca.com 8443

Trying 75.126.0.63...

telnet: connect to address 75.126.0.63: Connection timed out

So is there anyway we can use the images from local and make it run their containers run?

2. http://<hostname>:<port>/devtest and in that example host name is container name?
3. Will docker pull command store the images in Local system?, if yes where it will store.?
       I did not find pulled images in my windows box anywhere. 

[Giridharan Jagadeesan]

#1 - Ok. Actually, you do not need internet connection. Ensure if your have run docker pull correctly on your Linux machine. Then run docker ps -a to check the status. If the pull is successful, and when you run installSVDocker.sh file, the error will not to appear. I suspect that the images are not pulled correctly. Pls check this once.
#2 - No, hostname is where you run the containers - FQDN or IP address of your linux machine.
#3 - You can run "docker images" to find where it is stored. 

Cheers,
Giri​​​

[Saikiran Garapati]

Thanks for the quick reply I tried docker pull command in Linux machine 
and i got below error.

docker pull sv-docker.packages.ca.com/sv/config-server
Using default tag: latest
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/create?fromImage=sv-docker.packages.ca.com%2Fsv%2Fconfig-server&tag=latest: dial unix /var/run/docker.sock: connect: permission denied
 sudo docker pull sv-docker.packages.ca.com/sv/config-server
Using default tag: latest
Error response from daemon: Get https://sv-docker.packages.ca.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
 docker pull sv-docker.packages.ca.com/sv/iamm
Using default tag: latest
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.40/images/create?fromImage=sv-docker.packages.ca.com%2Fsv%2Fiamm&tag=latest: dial unix /var/run/docker.sock: connect: permission denied
 sudo su docker pull sv-docker.packages.ca.com/sv/iamm
su: user docker does not exist
sudo docker pull sv-docker.packages.ca.com/sv/iamm
Using default tag: latest
Error response from daemon: Get https://sv-docker.packages.ca.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

docker image ls comman is not giving the location where it is stored

docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
gsk71911/bulletinboard 1.9 a4555d32ecc1 9 days ago 182MB
bulletinboard 1.9 a4555d32ecc1 9 days ago 182MB
gsk71911/docker101tutorial latest 6b5397fee046 12 days ago 26.5MB
docker101tutorial latest 6b5397fee046 12 days ago 26.5MB
<none> <none> c4cebf282bc5 12 days ago 121MB
<none> <none> a086a8311ac4 12 days ago 108MB
<none> <none> a24b6ad20ba4 12 days ago 220MB
sv-docker.packages.ca.com/sv/lisa latest 90acecfdf44f 12 days ago 2.19GB
sv-docker.packages.ca.com/sv/portal latest 7bf1ff486d42 13 days ago 1.83GB
python alpine 8ecf5a48c789 13 days ago 78.9MB
node 12-alpine 1c342643aa5c 2 weeks ago 89.4MB
node current-slim 26379995b148 2 weeks ago 165MB
nginx alpine 7d0cdcc60a96 2 weeks ago 21.3MB
sv-docker.packages.ca.com/sv/virtual-service-catalog latest 874e144fa075 2 weeks ago 484MB
sv-docker.packages.ca.com/sv/config-server latest b3a16ec0be4b 2 weeks ago 347MB
sv-docker.packages.ca.com/sv/iaam latest e705d8b9b416 2 weeks ago 771MB
ubuntu latest 1d622ef86b13 7 weeks ago 73.9MB
docker/getting-started latest 3c156928aeec 8 weeks ago 24.8MB
docker/desktop-storage-provisioner v1.1 e704287ce753 2 months ago 41.8MB
docker/desktop-vpnkit-controller v1.0 79da37e5a3aa 3 months ago 36.6MB
docker/desktop-kubernetes kubernetes-v1.16.5-cni-v0.7.5-critools-v1.15.0 a86647f0b376 4 months ago 279MB
k8s.gcr.io/kube-apiserver v1.16.5 fc838b21afbb 5 months ago 159MB
k8s.gcr.io/kube-scheduler v1.16.5 b4d073a9efda 5 months ago 83.5MB
k8s.gcr.io/kube-controller-manager v1.16.5 441835dd2301 5 months ago 151MB
k8s.gcr.io/kube-proxy v1.16.5 0ee1b8a3ebe0 5 months ago 82.7MB
hello-world latest bf756fb1ae65 5 months ago 13.3kB
docker/kube-compose-controller v0.4.25-alpha1 129151cdf35f 7 months ago 35.6MB
docker/kube-compose-api-server v0.4.25-alpha1 989749268895 7 months ago 50.7MB
docker/kube-compose-installer v0.4.25-alpha1 2a71ac5a1359 7 months ago 42.3MB
k8s.gcr.io/etcd 3.3.15-0 b2756210eeab 9 months ago 247MB
k8s.gcr.io/coredns 1.6.2 bf261d157914 10 months ago 44.1MB
k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB

[Giridharan Jagadeesan]

Hi Saikiran,

Have you got the credentials from Support to access the Bintray repo? As stated in the documentation - Artifacts section, you need this. I have sent it privately to you as it contains password. This first step should resolve your permission issue hopefully. Step 2 is docker pull command.

Regarding the location where the images are stored; am not sure thought docker images would provide. Will share if I find out.

Cheers,
Giri

[Saikiran Garapati]

Hi Giridharan,

We got the credentials from Support team ad able to access the same in Windows system and able to pull the images also but in Linux machine since we do not have Internet connection, We are looking for an other option as it is difficult to get internet connection in Linux box.

If Pulled images are not stored in windows system also, I do not know how to achieve this without internet in Linux box.

Please suggest us if you have alternate solutions for this problem.

Thanks & Regards,
Saikiran Garapati,
08148860968

[Surya Anil Kumar Suravarapu]

Hello Saikiran,
            If you were able to pull the images in the windows system, you can save the image as a .tar file, move it to the linux server and then use docker load to load the images there. Please find information on docker save and docker load below.

https://docs.docker.com/engine/reference/commandline/save/
https://docs.docker.com/engine/reference/commandline/load/

Regards,
Surya

------------------------------
Client Services Consultant
Broadcom
------------------------------

[Saikiran Garapati]

Hi Surya,

Thanks for the reply. We were trying the same but we are facing problems with lisa.tar as it is having more than 2GB.

Remaining 4 images we saved using docker save command and moved it to Linux box using Winscp and docker load works fine i am able to see docker images are listed after giving docker images ls

But lisa.image is having more than 2GB size and I am not able to transfer it through winscp and for alternate solution we are transferring it to internal file system and downloading it in linux box after downloading when we give docker load command it is giving below error.

Error processing tar file(exit status 1): archive/tar: invalid tar header

Please let me know do we have any alternate solution for this..

[Saikiran Garapati]

Hi Team,

We managed to transfer the lisa image to Linux box and docker load succeed and docker image ls command is showing all docker images in the system.

docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
sv-docker.packages.ca.com/sv/lisa latest 90acecfdf44f 2 weeks ago 2.19GB
sv-docker.packages.ca.com/sv/portal latest 7bf1ff486d42 2 weeks ago 1.83GB
sv-docker.packages.ca.com/sv/virtual-service-catalog latest 874e144fa075 3 weeks ago 484MB
sv-docker.packages.ca.com/sv/config-server latest b3a16ec0be4b 3 weeks ago 347MB
sv-docker.packages.ca.com/sv/iaam latest e705d8b9b416 3 weeks ago 771MB

Is it mandatory to have internet connection for the Linux box where ever dockers images are stored to run ./installSVDocker.sh -d file?

My understanding is if Docker images are available locally, it will no need to pull it again from the Repository but when i execute installSVDocker.sh file again it is trying to pull it from the (sv-docker.packages.ca.com/sv/config-server:0.0.2) and since we do not internet connectivity it is failing.

after running below we are getting below error:

sudo ./installSVDocker.sh -d https://xxxxxxxxxxxx:15xx
Prerequisite checks
Checking for Docker
Docker is already installed
Docker version is 19.03
Check for Docker-Compose
docker-compose is already installed
docker-compose version 1.26.0, build d4451659
Checks if Enterprise Dashboard is accessible and other prerequisite checks
The provided enterprise dashboard https://xxxxxxxxxx:15xx is up.
Creating User and Group with name devtest and id 1010 if not present
Useful Information regarding VSEs
Virtual Services should start in below range for Virtual Service Environment - vse
20000-20049
Installation Directory is:
Pulling Images
Creating lisa-network and starting containers
Pulling config-server (sv-docker.packages.ca.com/sv/config-server:0.0.2)...
ERROR: Get https://sv-docker.packages.ca.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4948a2a6d983 sv-docker.packages.ca.com/sv/virtual-service-catalog "/bin/sh -c ./docker…" 3 minutes ago Up 3 minutes 8080/tcp admiring_galileo
c555a25ede24 sv-docker.packages.ca.com/sv/portal "/bin/sh -c ./docker…" 4 minutes ago Up 4 minutes 1507/tcp pensive_ellis
93c6dfb282c1 sv-docker.packages.ca.com/sv/lisa "sh /devtest/docker-…" 5 minutes ago Up 5 minutes 1505/tcp, 1528/tcp, 2009-2011/tcp, 2013-2014/tcp stoic_sanderson
3fdef69cae8a sv-docker.packages.ca.com/sv/config-server "java -Djava.securit…" 7 minutes ago Up 7 minutes 8888/tcp strange_diffie
fa5a43d2421b sv-docker.packages.ca.com/sv/lisa "sh /devtest/docker-…" 26 minutes ago Exited (130) 19 minutes ago relaxed_johnson
39c71bb7461a sv-docker.packages.ca.com/sv/config-server "java -Djava.securit…" 28 minutes ago Exited (130) 22 minutes ago mystifying_solomon
2040473ccfca sv-docker.packages.ca.com/sv/iaam "sh docker-entrypoin…" 4 days ago Up 4 days wizardly_hermann
22d41f9d9c05 sv-docker.packages.ca.com/sv/iaam "sh docker-entrypoin…" 4 days ago Up 4 days trusting_greider

Even after performing docker run command also it is (executing /installSVDocker.sh -d) trying to pull the images from Repository..

Please let us know any alternate solution if we have any running that command and going to pull the image from repository as it is very difficult to get the internet connection to Linux box?

[Surya Anil Kumar Suravarapu]

Hello Saikiran,

           Please try to pull the image - sv-docker.packages.ca.com/sv/config-server:0.0.2 on your windows machine and copy that image to the linux server. Once done, please try to run the command again. This may prevent the component from making call to pull images from the repo. Alternatively, you also have the option to update the docker registry to a private docker repo at your end by updating the .env file (property - SV_DOCKER_REGISTRY). Please try and let me know. 

Regards,
Surya

------------------------------
Client Services Consultant
Broadcom
------------------------------

[Saikiran Garapati]

Hi Surya,

I tried to save that image in my local windows but it says response from daemon: reference does not exist

C:\XXXXXX\XXXXXX>docker save -o C:\XXXXXXXXXXXXX\Desktop\TmobileSV\StepListsSanityCasesAndImpLinks\K8S\Docker\onlyConfigServer/config-server0.0.2.tar sv-docker.packages.ca.com/sv/config-server:0.0.2
Error response from daemon: reference does not exist

and afterwards  tried without :0.0.2 then it saved in windows and Moved it to Linux and ran a .installSV Docker.sh file but same error.

sudo ./installSVDocker.sh -d https://xxxxxxxxxxxxxxxxxxxxxx:1506
Prerequisite checks
Checking for Docker
Docker is already installed
Docker version is 19.03
Check for Docker-Compose
docker-compose is already installed
docker-compose version 1.26.0, build d4451659
Checks if Enterprise Dashboard is accessible and other prerequisite checks
The provided enterprise dashboard https://xxxxxxxxxxxxxxxxxxxxx:1506 is up.
Creating User and Group with name devtest and id 1010 if not present
Useful Information regarding VSEs
Virtual Services should start in below range for Virtual Service Environment - vse
20000-20049
Installation Directory is:
Pulling Images
Creating lisa-network and starting containers
Pulling config-server (sv-docker.packages.ca.com/sv/config-server:0.0.2)...
ERROR: Get https://sv-docker.packages.ca.com/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

When you say pull can i consider that one as a docker save because after docker pull , that image will store in hard disk image and we can not trasfer that hard disk image file in to Linux i believe. Please correct me if i am wrong.

.

[Surya Anil Kumar Suravarapu]

Hello Saikiran,

            If the images are up to date, you will get the flow as given below. I suspect the docker image for config server 0.0.2 didn't make it to your server. Please double check if the image is available and try again. We can have a webex session to go over this as needed. Please email me in case you face any challenges. 

Regards,
Surya

------------------------------
Client Services Consultant
Broadcom
------------------------------

[Surya Anil Kumar Suravarapu]

Just to keep others posted, once the images including config-server were migrated, Saikiran confirmed that they are able to bring up the components using docker compose. 

Regards,
Surya

------------------------------
Client Services Consultant
Broadcom
------------------------------

[Saikiran Garapati]

Hi Team,

Thank you very much for your extended support.
Can we configure multiple Portals? 

I have changed limits.env and ports.env file and executed installSvDocker file

limits.env

VSE_LIMIT=10
COORDINATOR_LIMIT=10
SIMULATOR_LIMIT=10
PORTAL_LIMIT=3

ports.env

PORTAL1_PORT=1508
PORTAL2_PORT=1509
PORTAL3_PORT=1510

Error:

./installSVDocker.sh -d http://XXXXXXXXXXXXXXXXXX:1506 --portals 3
./installSVDocker.sh: unrecognized option '--portals'
Enter './installSVDocker.sh --help' for more information


Is there any way we can configure multiple Portals for the same script?

I can ignore the below error as -P is for SSL options:

./installSVDocker.sh -d http://XXXXXXXXXXXXXXXXXXXXXX:1506 -P 3
Prerequisite checks
Checking for Docker
Docker is already installed
Docker version is 19.03
Check for Docker-Compose
docker-compose is already installed
docker-compose version 1.26.0, build d4451659
Checks if Enterprise Dashboard is accessible and other prerequisite checks
Please enable SSL mode to use these options
Try './installSVDocker --help' for more information

[Surya Anil Kumar Suravarapu]

Hello Saikiran,

          What is the use case to have multiple portals? A single portal can show the details of all the resources connected to the registry. Please let me know. 

Regards,
Surya

------------------------------
Client Services Consultant
Broadcom
------------------------------

[Saikiran Garapati]

Hi Surya,

We do have Multiple teams for Testing and as well as in VS side for each environment and as well as different environments.
Please let us also know can we have multiple registries also..