Image Processing Archives - Robotics with ROS

Pose estimation and tracking human is one the key step in sports analysis. Here is in this work I used openpose for analysis of player in a Bundesliga game HSV Hamburg vs Bayer München. Warning: the video might be disturbing for HSV fans 🙂 Original Video Analyzed Video Original Video Analyzed Video Original Video Analyzed Video Original Video

Human detection and Pose Estimation with Deep Learning for Sport Analysis Read More »

RGBD PCL point cloud from Stereo vision with ROS and OpenCV

Leave a Comment / Computer Vision, Image Processing, ROS, Tutorials / admin

In my other tutorial, I showed you how to calibrate you stereo camera. After Calibration, we can get disparity map and RGBD PCL point cloud from our stereo camera cool huh 🙂 1)Save the following text under “stereo_usb_cam_stream_publisher.launch”

<launch>

<arg name="video_device_right" default="/dev/video1" />
<arg name="video_device_left" default="/dev/video2" />

<arg name="image_width" default="640" />
<arg name="image_height" default="480" />

<arg name="right_image_namespace" default="stereo" />
<arg name="left_image_namespace" default="stereo" />

<arg name="right_image_node" default="right" />
<arg name="left_image_node" default="left" />

<arg name="image_topic_name_right" default="image_raw" />
<arg name="camera_info_topic_name_right" default="camera_info" />

<arg name="image_topic_name_left" default="image_raw" />
<arg name="camera_info_topic_name_left" default="camera_info" />



<arg name="left_camera_name" default="left" />
<arg name="right_camera_name" default="right" />

<arg name="left_camera_frame_id" default="left_camera" />
<arg name="right_camera_frame_id" default="right_camera" />

<arg name="framerate" default="30" />

<arg name="camera_info" default="false" />
<arg name="camera_info_path" default="stereo_camera_info" />
	  
<node ns="$(arg right_image_namespace)" name="$(arg right_image_node)" pkg="usb_cam" type="usb_cam_node" output="screen" >
	<param name="video_device" value="$(arg video_device_right)" />
	<param name="image_width" value="$(arg image_width)" />
	<param name="image_height" value="$(arg image_height)"/>
	<param name="pixel_format" value="yuyv" />
	<param name="io_method" value="mmap"/>
	<remap from="/usb_cam/image_raw" to="$(arg image_topic_name_right)" />
	<param name="framerate" value="$(arg framerate)"/> 
	<!-- if camera_info is available, we will use it-->
	<param name="camera_frame_id" value="$(arg right_camera_frame_id)" if="$(arg camera_info)" /> 
	<param name="camera_info_url" value="file://${ROS_HOME}/$(arg camera_info_path)/right.yaml" if="$(arg camera_info)"/>
	<param name="camera_name" value="narrow_stereo/$(arg right_camera_name)" if="$(arg camera_info)"/> 
	<remap from="/usb_cam/camera_info" to="$(arg camera_info_topic_name_right)" if="$(arg camera_info)"/>

</node>

<node ns="$(arg left_image_namespace)" name="$(arg left_image_node)" pkg="usb_cam" type="usb_cam_node" output="screen" >
	<param name="video_device" value="$(arg video_device_left)" />
	<param name="image_width" value="$(arg image_width)" />
	<param name="image_height" value="$(arg image_height)"/>
	<param name="pixel_format" value="yuyv" />
	<param name="io_method" value="mmap"/>
	<param name="framerate" value="$(arg framerate)"/> 
	<remap from="/usb_cam/image_raw" to="$(arg image_topic_name_left)"/>
	<!-- if camera_info is available, we will use it-->
	<param name="camera_frame_id" value="$(arg left_camera_frame_id)" if="$(arg camera_info)"/> 
	<param name="camera_name" value="narrow_stereo/$(arg left_camera_name)" if="$(arg camera_info)"/> 
	<param name="camera_info_url" value="file://${ROS_HOME}/$(arg camera_info_path)/left.yaml" if="$(arg camera_info)"/> 
	<remap from="/usb_cam/camera_info" to="$(arg camera_info_topic_name_left)" if="$(arg camera_info)"/>
</node>
</launch>

</node>

</node>

</launch>

2) Then run the following node to publish both cameras and camera info (calibration matrix)

roslaunch stereo_usb_cam_stream_publisher.launch video_device_right:=/dev/video1 video_device_left:=/dev/video2 image_width:=640 image_height:=480 left_camera_name:=left right_camera_name:=right camera_info:=true camera_info_path:=stereo_camera_info

1	roslaunch stereo_usb_cam_stream_publisher.launch video_device_right:=/dev/video1 video_device_left:=/dev/video2 image_width:=640 image_height:=480 left_camera_name:=left right_camera_name:=right camera_info:=true camera_info_path:=stereo_camera_info

3) Run the

RGBD PCL point cloud from Stereo vision with ROS and OpenCV Read More »

Stereo Camera Calibration with ROS and OpenCV

5 Comments / Computer Vision, Image Processing, ROS, Tutorials / admin

In this tutorial, I’m gonna show you stereo camera calibration with ROS and OpenCV. So you need a pair of cameras, I bought a pair of this USB webcam which is okay for this task. 1)Save the following text under “stereo_usb_cam_stream_publisher.launch”

<launch>

<arg name="video_device_right" default="/dev/video1" />
<arg name="video_device_left" default="/dev/video2" />

<arg name="image_width" default="640" />
<arg name="image_height" default="480" />

<arg name="right_image_namespace" default="stereo" />
<arg name="left_image_namespace" default="stereo" />

<arg name="right_image_node" default="right" />
<arg name="left_image_node" default="left" />

<arg name="image_topic_name_right" default="image_raw" />
<arg name="camera_info_topic_name_right" default="camera_info" />

<arg name="image_topic_name_left" default="image_raw" />
<arg name="camera_info_topic_name_left" default="camera_info" />



<arg name="left_camera_name" default="left" />
<arg name="right_camera_name" default="right" />

<arg name="left_camera_frame_id" default="left_camera" />
<arg name="right_camera_frame_id" default="right_camera" />

<arg name="framerate" default="30" />

<arg name="camera_info" default="false" />
<arg name="camera_info_path" default="stereo_camera_info" />
	  
<node ns="$(arg right_image_namespace)" name="$(arg right_image_node)" pkg="usb_cam" type="usb_cam_node" output="screen" >
	<param name="video_device" value="$(arg video_device_right)" />
	<param name="image_width" value="$(arg image_width)" />
	<param name="image_height" value="$(arg image_height)"/>
	<param name="pixel_format" value="yuyv" />
	<param name="io_method" value="mmap"/>
	<remap from="/usb_cam/image_raw" to="$(arg image_topic_name_right)" />
	<param name="framerate" value="$(arg framerate)"/> 
	<!-- if camera_info is available, we will use it-->
	<param name="camera_frame_id" value="$(arg right_camera_frame_id)" if="$(arg camera_info)" /> 
	<param name="camera_info_url" value="file://${ROS_HOME}/$(arg camera_info_path)/right.yaml" if="$(arg camera_info)"/>
	<param name="camera_name" value="narrow_stereo/$(arg right_camera_name)" if="$(arg camera_info)"/> 
	<remap from="/usb_cam/camera_info" to="$(arg camera_info_topic_name_right)" if="$(arg camera_info)"/>

</node>

<node ns="$(arg left_image_namespace)" name="$(arg left_image_node)" pkg="usb_cam" type="usb_cam_node" output="screen" >
	<param name="video_device" value="$(arg video_device_left)" />
	<param name="image_width" value="$(arg image_width)" />
	<param name="image_height" value="$(arg image_height)"/>
	<param name="pixel_format" value="yuyv" />
	<param name="io_method" value="mmap"/>
	<param name="framerate" value="$(arg framerate)"/> 
	<remap from="/usb_cam/image_raw" to="$(arg image_topic_name_left)"/>
	<!-- if camera_info is available, we will use it-->
	<param name="camera_frame_id" value="$(arg left_camera_frame_id)" if="$(arg camera_info)"/> 
	<param name="camera_name" value="narrow_stereo/$(arg left_camera_name)" if="$(arg camera_info)"/> 
	<param name="camera_info_url" value="file://${ROS_HOME}/$(arg camera_info_path)/left.yaml" if="$(arg camera_info)"/> 
	<remap from="/usb_cam/camera_info" to="$(arg camera_info_topic_name_left)" if="$(arg camera_info)"/>
</node>
</launch>

</node>

</node>

</launch>

2)Then run the following node to publish both cameras.

roslaunch stereo_usb_cam_stream_publisher.launch video_device_right:=/dev/video1 video_device_left:=/dev/video2 image_width:=640 image_height:=480 left_camera_name:=left right_camera_name:=right

1	roslaunch stereo_usb_cam_stream_publisher.launch video_device_right:=/dev/video1 video_device_left:=/dev/video2 image_width:=640 image_height:=480 left_camera_name:=left right_camera_name:=right

3)Now call the calibration node:

Stereo Camera Calibration with ROS and OpenCV Read More »

Open source Structure-from-Motion and Multi-View Stereo tools with C++

Leave a Comment / C++, Computer Vision, Image Processing, Linear Algebra, Tutorials / admin

Structure-from-Motion (SFM) is genuinely an interesting topic in computer vision, Basically making 3D structure from something 2D is absolutely mesmerizing 🙂 There two open source yet very robust tools for SFM, which sometimes compiling them might be complicated, here I will share my experience with you. 1)VisualSFM Prerequisite: 1)Glew Download the glew from SF at http://glew.sourceforge.net/. Do NOT get it from Github as

Open source Structure-from-Motion and Multi-View Stereo tools with C++ Read More »

Finding Homography Matrix using Singular-value Decomposition and RANSAC in OpenCV and Matlab

3 Comments / C++, Computer Vision, Image Processing, Linear Algebra, Matlab, Tutorials / admin

Solving a Homography problem leads to solving a set of homogeneous linear equations such below: \begin{equation} \left( \begin{array}{ccccccccc} -x1 & -y1 & -1 & 0 & 0 & 0 & x1*xp1 & y1*xp1 & xp1\\ 0 & 0 & 0 & -x1 & -y1 & -1 & x1*yp1 & y1*yp1 & yp1\\ -x2 & -y2 &

Finding Homography Matrix using Singular-value Decomposition and RANSAC in OpenCV and Matlab Read More »

Finding Affine Transform with Linear Least Squares

3 Comments / Computer Vision, Image Processing, Linear Algebra, Tutorials / admin

linear least squares is a method of fitting a model to data in which the relation between data and unknown paramere can be expressed in a linear form. $ Ax=b$ $ X^{*}= \underset{x}{\mathrm{argmin}}= \|Ax-b\|^{2} =(A^{T}A)^{-1}A^{T}b $

function M= affine_least_square(x0,y0, x1,y1, x2,y2, xp0,yp0, xp1,yp1, xp2,yp2)
% This function finds the affine transform between three points

% an affine transformation with a 3x3 affine transformation matrix: 
% 
% [M11 M12 M13]
% [M21 M22 M23]
% [M31 M32 M33]

%Since the third row is always [0 0 1] we can skip that.

% [x0 y0 1  0  0  0 ]     [M11]   [xp0]
% [0  0  0  x0 y0 1 ]     [M12]   [yp0] 
% [x1 y1 1  0  0  0 ]     [M13]   [xp1]
% [0  0  0  x1 y1 1 ]  *  [M21] = [yp1] 
% [x2 y2 1  0  0  0 ]     [M22]   [xp2] 
% [0  0  0  x2 y2 1 ]     [M23]   [yp2]
                          
%         A            *    X   =  B
%A -> 6x6
%X -> 6x1 in fact 
%X=A\B


A=[x0 y0 1  0  0  0 ; 0  0  0  x0 y0 1 ; x1 y1 1  0  0  0 ; 0  0  0  x1 y1 1 ; x2 y2 1  0  0  0;0  0  0  x2 y2 1];
B=[xp0; yp0; xp1;  yp1; xp2 ; yp2 ];


% X=A\B; 
% this is what we need but accroding to https://www.mathworks.com/matlabcentral/newsreader/view_thread/170201
%The following is better:
X=pinv(A)*B;

M11 =X(1,1);
M12 =X(2,1);  
M13 =X(3,1);
M21 =X(4,1);
M22 =X(5,1);
M23 =X(6,1);
M=[ M11 M12 M13; M21 M22 M23; 0 0 1];
end

function M= affine_least_square(x0,y0, x1,y1, x2,y2, xp0,yp0, xp1,yp1, xp2,yp2)

% This function finds the affine transform between three points

% an affine transformation with a 3x3 affine transformation matrix:

% [M11 M12 M13]

% [M21 M22 M23]

% [M31 M32 M33]

%Since the third row is always [0 0 1] we can skip that.

% [x0 y0 1 0 0 0 ] [M11] [xp0]

% [0 0 0 x0 y0 1 ] [M12] [yp0]

% [x1 y1 1 0 0 0 ] [M13] [xp1]

% [0 0 0 x1 y1 1 ] * [M21] = [yp1]

% [x2 y2 1 0 0 0 ] [M22] [xp2]

% [0 0 0 x2 y2 1 ] [M23] [yp2]

% A * X = B

%A -> 6x6

%X -> 6x1 in fact

%X=A\B

A=[x0 y0 1 0 0 0 ; 0 0 0 x0 y0 1 ; x1 y1 1 0 0 0 ; 0 0 0 x1 y1 1 ; x2 y2 1 0 0 0;0 0 0 x2 y2 1];

B=[xp0; yp0; xp1; yp1; xp2 ; yp2 ];

% X=A\B;

% this is what we need but accroding to https://www.mathworks.com/matlabcentral/newsreader/view_thread/170201

%The following is better:

X=pinv(A)*B;

M11 =X(1,1);

M12 =X(2,1);

M13 =X(3,1);

M21 =X(4,1);

M22 =X(5,1);

M23 =X(6,1);

M=[ M11 M12 M13; M21 M22 M23; 0 0 1];

end

And testing the code:

clc;
clear all;
%This is a simple test that project 3 point with an affine trasnform and
%then tries to find the transformation:
% tform.T and M' should be equal

%Points are located at:
x0=0;
y0=0; 
x1=1;
y1=0; 
x2=0;
y2=1;

p1=[x0,y0; x1,y1; x2,y2];

%We rotate them (a roll-rotation around Z axes) theta degree, first
%building the rotation matrix
theta=24;
A11=cosd(theta);
A12=-sind(theta);
A21=sind(theta);
A22=cosd(theta);
%Translation part
Tx=1;
Ty=2;

tform = affine2d([A11 A12 0; A21 A22 0; Tx Ty 1]);
fprintf('3x3 transformation matrix:')
tform.T

[x,y]=transformPointsForward(tform,p1(:,1),p1(:,2));
fprintf('transformed points:')
[x,y]


xp0=x(1,1);
yp0=y(1,1);
xp1=x(2,1);
yp1=y(2,1);
xp2=x(3,1);
yp2=y(3,1);



%finding the tranformation matrix from set of points and their respective
%transformed points:
M=affine_least_square(x0,y0, x1,y1, x2,y2, xp0,yp0, xp1,yp1, xp2,yp2);
fprintf('3x3 affine transformation matrix from least_squares:')
M'

tform = affine2d(M');
[x,y]=transformPointsForward(tform,p1(:,1),p1(:,2));
[x,y];

clc;

clear all;

%This is a simple test that project 3 point with an affine trasnform and

%then tries to find the transformation:

% tform.T and M' should be equal

%Points are located at:

x0=0;

y0=0;

x1=1;

y1=0;

x2=0;

y2=1;

p1=[x0,y0; x1,y1; x2,y2];

%We rotate them (a roll-rotation around Z axes) theta degree, first

%building the rotation matrix

theta=24;

A11=cosd(theta);

A12=-sind(theta);

A21=sind(theta);

A22=cosd(theta);

%Translation part

Tx=1;

Ty=2;

tform = affine2d([A11 A12 0; A21 A22 0; Tx Ty 1]);

fprintf('3x3 transformation matrix:')

tform.T

[x,y]=transformPointsForward(tform,p1(:,1),p1(:,2));

fprintf('transformed points:')

[x,y]

xp0=x(1,1);

yp0=y(1,1);

xp1=x(2,1);

yp1=y(2,1);

xp2=x(3,1);

yp2=y(3,1);

%finding the tranformation matrix from set of points and their respective

%transformed points:

M=affine_least_square(x0,y0, x1,y1, x2,y2, xp0,yp0, xp1,yp1, xp2,yp2);

fprintf('3x3 affine transformation matrix from least_squares:')

tform = affine2d(M');

[x,y]=transformPointsForward(tform,p1(:,1),p1(:,2));

[x,y];

3x3 transformation matrix:
ans =

    0.9135   -0.4067         0
    0.4067    0.9135         0
    1.0000    2.0000    1.0000

transformed points:
ans =

    1.0000    2.0000
    1.9135    1.5933
    1.4067    2.9135

3x3 affine transformation matrix from least_squares:
ans =

    0.9135   -0.4067         0
    0.4067    0.9135         0
    1.0000    2.0000    1.0000

3x3 transformation matrix:

ans =

0.9135 -0.4067 0

0.4067 0.9135 0

1.0000 2.0000 1.0000

transformed points:

ans =

1.0000 2.0000

1.9135 1.5933

1.4067 2.9135

3x3 affine transformation matrix from least_squares:

ans =

0.9135 -0.4067 0

0.4067 0.9135 0

1.0000 2.0000 1.0000

Finding Affine Transform with Linear Least Squares Read More »

Peak Signal-to-Noise Ratio (PSNR) in Image using OpenCV and Matlab

Leave a Comment / C++, Computer Vision, Image Processing, Matlab, Tutorials / admin

Peak signal-to-noise ratio (PSNR) shows the ratio between the maximum possible power of a signal and the power of the same image with noise. PSNR is usually expressed in logarithmic decibel scale. $ MSE =1/m*n \sum_{i=0}^{m-1} \sum_{j=0}^{n-1} [ Image( i,j) -NoisyImage( i,j) ] ^2 $ $ PSNR =10* \log_{10} \Bigg(MAXI^2/MSE\Bigg) $ MSE is Mean Square Error and MAXI is

Peak Signal-to-Noise Ratio (PSNR) in Image using OpenCV and Matlab Read More »

Seam Carving Algorithm for Content-Aware Image Resizing with Matlab Code

1 Comment / Computer Vision, Image Processing, Machine Learning, Tutorials / admin

Seam carving is an algorithm for resizing images while keeping the most prominent and conspicuous pixels in the image. The important pixels in an image are usually those who are located over horizontal or vertical edges, so to throw away some pixels we first find horizontal and vertical edges and store their magnitude as pixel

Seam Carving Algorithm for Content-Aware Image Resizing with Matlab Code Read More »

HARRIS Corner detector explained

4 Comments / Computer Vision, Image Processing, Linear Algebra, Machine Learning, Tutorials / admin

im = imread('chessboard.jpg');
im = double(rgb2gray(im))/255;

sigma = 5;
g = fspecial('gaussian', 2*sigma*3+1, sigma);
dx = [-1 0 1;-1 0 1; -1 0 1];
Ix = imfilter(im, dx, 'symmetric', 'same');
Iy = imfilter(im, dx', 'symmetric', 'same');
Ix2 = imfilter(Ix.^2, g, 'symmetric', 'same');
Iy2 = imfilter(Iy.^2, g, 'symmetric', 'same');
Ixy = imfilter(Ix.*Iy, g, 'symmetric', 'same');
k = 0.04; % k is between 0.04 and 0.06
% --- r = Det(M) - kTrace(M)^2 ---
r = (Ix2.*Iy2 - Ixy.^2) - k*(Ix2 + Iy2).^2;

norm_r = r - min(r(:));
norm_r = norm_r ./ max(norm_r(:));
subplot(1,2,1), imshow(im)
subplot(1,2,2), imshow(norm_r)


%hLocalMax = vision.LocalMaximaFinder;
%hLocalMax.MaximumNumLocalMaxima = 100;
%hLocalMax.NeighborhoodSize = [3 3];
%hLocalMax.Threshold = 0.05;
%location = step(hLocalMax, norm_r);

im = imread('chessboard.jpg');

im = double(rgb2gray(im))/255;

sigma = 5;

g = fspecial('gaussian', 2*sigma*3+1, sigma);

dx = [-1 0 1;-1 0 1; -1 0 1];

Ix = imfilter(im, dx, 'symmetric', 'same');

Iy = imfilter(im, dx', 'symmetric', 'same');

Ix2 = imfilter(Ix.^2, g, 'symmetric', 'same');

Iy2 = imfilter(Iy.^2, g, 'symmetric', 'same');

Ixy = imfilter(Ix.*Iy, g, 'symmetric', 'same');

k = 0.04; % k is between 0.04 and 0.06

% --- r = Det(M) - kTrace(M)^2 ---

r = (Ix2.*Iy2 - Ixy.^2) - k*(Ix2 + Iy2).^2;

norm_r = r - min(r(:));

norm_r = norm_r ./ max(norm_r(:));

subplot(1,2,1), imshow(im)

subplot(1,2,2), imshow(norm_r)

%hLocalMax = vision.LocalMaximaFinder;

%hLocalMax.MaximumNumLocalMaxima = 100;

%hLocalMax.NeighborhoodSize = [3 3];

%hLocalMax.Threshold = 0.05;

%location = step(hLocalMax, norm_r);

HARRIS Corner detector explained Read More »

Cookie	Duration	Description
cookielawinfo-checbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.